Skip to content

fix: remove apiKey tokens from SKILL.md to prevent LLM priming (#1696) - #1704

Merged
Jon Gallant (jongio) merged 1 commit into
mainfrom
fix/byom-apikey-positive-only-1696
Apr 3, 2026
Merged

fix: remove apiKey tokens from SKILL.md to prevent LLM priming (#1696)#1704
Jon Gallant (jongio) merged 1 commit into
mainfrom
fix/byom-apikey-positive-only-1696

Conversation

@jongio

@jongio Jon Gallant (jongio) commented Apr 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Fixes #1696

Summary

PR #1685 (merged yesterday) tried to fix the BYOM apiKey leak by adding "NEVER use apiKey" warnings to SKILL.md. This backfired — mentioning apiKey, AZURE_OPENAI_API_KEY, and AZURE_OPENAI_KEY in the skill context primed the LLM to emit those exact tokens during generation.

Root Cause

LLMs use token frequency for generation. Negative examples ("NEVER use apiKey") still inject apiKey into the model's context window, reinforcing the pattern. The reference files (azure-model-config.md, etc.) were already clean — the SKILL.md warnings were the source of contamination.

Why PR #1685 Wasn't Caught

Gap Detail
No integration tests PR ran unit tests (42/42) but integration tests require real agent runs and weren't triggered
Waza eval is mock-only executor: mock doesn't invoke the real agent
Waza eval didn't check apiKey byom-config.yaml had output_not_contains: [error, failed] but no apiKey check

Fix

File Change
SKILL.md Removed ALL apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY mentions. Replaced with positive-only guidance: "ONLY bearerToken via DefaultAzureCredential". Version 1.0.4 to 1.0.5
byom-config.yaml Added apiKey, AZURE_OPENAI_API_KEY, AZURE_OPENAI_KEY to output_not_contains

Verification

  • grep -ri for apiKey patterns in plugin/skills/azure-hosted-copilot-sdk/ returns zero matches
  • 42/42 unit + trigger tests pass
  • No lint/type errors

Testing

The real validation requires an integration test run against this branch. The BYOM content-quality test (countApiKeyInByomConfig === 0) is the definitive check.

@jongio
fix: remove apiKey tokens from SKILL.md to prevent LLM priming (#1696)
1328ee9 
PR #1685 tried to fix BYOM apiKey leak by adding 'NEVER use apiKey'
warnings. This backfired — mentioning apiKey in skill context primed the
model to emit it. Replace with positive-only auth guidance ('ONLY
bearerToken') and add apiKey absence checks to waza eval.

Changes:
- SKILL.md: Remove all apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY
  mentions, use positive-only auth framing, bump version 1.0.4 -> 1.0.5
- byom-config.yaml: Add apiKey patterns to output_not_contains

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 3, 2026 18:39
@github-actions

github-actions Bot commented Apr 3, 2026

Copy link
Copy Markdown
Contributor
Details# 🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric Value
📉 Total Change -24 tokens (-2%)
Before 990 tokens
After 966 tokens
Files Changed 1

Changed Files

File Before After Change
plugin/skills/azure-hosted-copilot-sdk/SKILL.md 990 966 -24 (-2%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown

📊 Token Limit Check Report

Checked: 536 files
Exceeded: 71 files

⚠️ Files Exceeding Token Limits

File Tokens Limit Over By
.github/skills/analyze-test-run/SKILL.md 2471 500 +1971
.github/skills/file-test-bug/SKILL.md 628 500 +128
.github/skills/sensei/README.md 3531 2000 +1531
.github/skills/sensei/SKILL.md 3026 500 +2526
.github/skills/sensei/references/EXAMPLES.md 3701 2000 +1701
.github/skills/sensei/references/LOOP.md 4181 2000 +2181
.github/skills/sensei/references/SCORING.md 4200 2000 +2200
.github/skills/skill-authoring/SKILL.md 839 500 +339
plugin/skills/appinsights-instrumentation/SKILL.md 908 500 +408
plugin/skills/azure-ai/SKILL.md 817 500 +317
plugin/skills/azure-aigateway/SKILL.md 1258 500 +758
plugin/skills/azure-aigateway/references/policies.md 2342 2000 +342
plugin/skills/azure-cloud-migrate/references/services/functions/lambda-to-functions.md 2600 2000 +600
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/javascript.md 2181 2000 +181
plugin/skills/azure-compliance/SKILL.md 1185 500 +685
plugin/skills/azure-compute/SKILL.md 755 500 +255
plugin/skills/azure-compute/workflows/vm-recommender/vm-recommender.md 2393 2000 +393
plugin/skills/azure-compute/workflows/vm-troubleshooter/references/cannot-connect-to-vm.md 7308 2000 +5308
plugin/skills/azure-cost/SKILL.md 1861 500 +1361
plugin/skills/azure-deploy/SKILL.md 1562 500 +1062
plugin/skills/azure-diagnostics/SKILL.md 1132 500 +632
plugin/skills/azure-diagnostics/aks-troubleshooting/networking.md 2147 2000 +147
plugin/skills/azure-diagnostics/aks-troubleshooting/node-issues.md 2003 2000 +3
plugin/skills/azure-enterprise-infra-planner/SKILL.md 991 500 +491
plugin/skills/azure-enterprise-infra-planner/references/constraints/compute-apps.md 2022 2000 +22
plugin/skills/azure-hosted-copilot-sdk/SKILL.md 966 500 +466
plugin/skills/azure-kubernetes/SKILL.md 2266 500 +1766
plugin/skills/azure-kusto/SKILL.md 2149 500 +1649
plugin/skills/azure-messaging/SKILL.md 967 500 +467
plugin/skills/azure-prepare/SKILL.md 2767 500 +2267
plugin/skills/azure-prepare/references/aspire.md 2991 2000 +991
plugin/skills/azure-prepare/references/plan-template.md 2559 2000 +559
plugin/skills/azure-prepare/references/recipes/azd/aspire.md 2101 2000 +101
plugin/skills/azure-prepare/references/recipes/azd/terraform.md 3012 2000 +1012
plugin/skills/azure-prepare/references/research.md 2217 2000 +217
plugin/skills/azure-prepare/references/resources-limits-quotas.md 3322 2000 +1322
plugin/skills/azure-prepare/references/security.md 2133 2000 +133
plugin/skills/azure-prepare/references/services/functions/bicep.md 3065 2000 +1065
plugin/skills/azure-prepare/references/services/functions/templates/SPEC-composable-templates.md 6187 2000 +4187
plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md 4649 2000 +2649
plugin/skills/azure-prepare/references/services/functions/terraform.md 3358 2000 +1358
plugin/skills/azure-quotas/SKILL.md 3445 500 +2945
plugin/skills/azure-quotas/references/commands.md 2644 2000 +644
plugin/skills/azure-resource-lookup/SKILL.md 1288 500 +788
plugin/skills/azure-resource-visualizer/SKILL.md 2054 500 +1554
plugin/skills/azure-storage/SKILL.md 1180 500 +680
plugin/skills/azure-upgrade/SKILL.md 1001 500 +501
plugin/skills/azure-upgrade/references/services/functions/automation.md 3463 2000 +1463
plugin/skills/azure-upgrade/references/services/functions/consumption-to-flex.md 2773 2000 +773
plugin/skills/azure-validate/SKILL.md 906 500 +406
plugin/skills/entra-app-registration/SKILL.md 2067 500 +1567
plugin/skills/entra-app-registration/references/api-permissions.md 2545 2000 +545
plugin/skills/entra-app-registration/references/cli-commands.md 2211 2000 +211
plugin/skills/entra-app-registration/references/console-app-example.md 2752 2000 +752
plugin/skills/entra-app-registration/references/oauth-flows.md 2375 2000 +375
plugin/skills/microsoft-foundry/SKILL.md 2870 500 +2370
plugin/skills/microsoft-foundry/foundry-agent/create/create.md 3016 2000 +1016
plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md 5511 2000 +3511
plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/eval-datasets.md 2342 2000 +342
plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/trace-to-dataset.md 4268 2000 +2268
plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md 2547 2000 +547
plugin/skills/microsoft-foundry/foundry-agent/trace/references/kql-templates.md 2701 2000 +701
plugin/skills/microsoft-foundry/models/deploy-model/SKILL.md 1640 500 +1140
plugin/skills/microsoft-foundry/models/deploy-model/capacity/SKILL.md 1739 500 +1239
plugin/skills/microsoft-foundry/models/deploy-model/customize/SKILL.md 2235 500 +1735
plugin/skills/microsoft-foundry/models/deploy-model/customize/references/customize-workflow.md 3335 2000 +1335
plugin/skills/microsoft-foundry/models/deploy-model/preset/SKILL.md 1226 500 +726
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/preset-workflow.md 5534 2000 +3534
plugin/skills/microsoft-foundry/quota/quota.md 2130 2000 +130
plugin/skills/microsoft-foundry/quota/references/capacity-planning.md 2080 2000 +80
plugin/skills/microsoft-foundry/references/sdk/foundry-sdk-py.md 2162 2000 +162

Consider moving content to references/ subdirectories.

Automated token analysis. See skill authoring guidelines for best practices.

Copilot AI reviewed Apr 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a regression where mentioning API-key tokens inside azure-hosted-copilot-sdk’s SKILL.md inadvertently primed the LLM to emit those tokens in generated BYOM config, causing integration test failures (Issue #1696). It removes those token mentions from the skill context while adding explicit negative assertions to the BYOM eval task to catch future regressions.

Changes:

  • Removed explicit API-key token mentions from the BYOM auth guidance in SKILL.md, replacing them with positive-only “bearerToken via DefaultAzureCredential/ManagedIdentityCredential” guidance.
  • Bumped azure-hosted-copilot-sdk skill version 1.0.4 → 1.0.5.
  • Strengthened the BYOM eval task by asserting the output must not contain API-key token strings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
plugin/skills/azure-hosted-copilot-sdk/SKILL.md Removes API-key token strings from skill context to avoid LLM priming; updates skill version.
evals/azure-hosted-copilot-sdk/tasks/byom-config.yaml Adds output_not_contains checks to prevent API-key token leakage in BYOM outputs.

@jongio
Jon Gallant (jongio) merged commit d12fd24 into main Apr 3, 2026
16 checks passed
@jongio
Jon Gallant (jongio) deleted the fix/byom-apikey-positive-only-1696 branch April 3, 2026 19:03
Barbara 4bes (Ba4bes) pushed a commit to Ba4bes/GitHub-Copilot-for-Azure that referenced this pull request Apr 24, 2026
@jongio
fix: remove apiKey tokens from SKILL.md to prevent LLM priming (micro…
2805b67 
…soft#1696) (microsoft#1704)

PR microsoft#1685 tried to fix BYOM apiKey leak by adding 'NEVER use apiKey'
warnings. This backfired — mentioning apiKey in skill context primed the
model to emit it. Replace with positive-only auth guidance ('ONLY
bearerToken') and add apiKey absence checks to waza eval.

Changes:
- SKILL.md: Remove all apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY
  mentions, use positive-only auth framing, bump version 1.0.4 -> 1.0.5
- byom-config.yaml: Add apiKey patterns to output_not_contains

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
+1 more reviewer
@kvenkatrajan kvenkatrajan kvenkatrajan approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Integration test failure: azure-hosted-copilot-sdk – BYOM DefaultAzureCredential API key leak [Assertion mismatch]

3 participants

@jongio @kvenkatrajan