fix: resolve azure-hosted-copilot-sdk test failures for timeout and BYOM API key leak (#1447, #1448) - #1685
Merged
Jon Gallant (jongio) merged 2 commits intoApr 2, 2026
Conversation
…ft#1447) Greenfield prompts trigger full planning + scaffolding per agent invocation (~6-8 min each). With RUNS_PER_PROMPT=3, the cumulative runtime exceeded the 20-min TEST_TIMEOUT. Introduce GREENFIELD_RUNS_PER_PROMPT=1 for the greenfield-explicit scenario, matching the pattern used by other slow suites (azure-deploy, azure-prepare). One run is sufficient to validate routing correctness for an explicit SDK mention.
…t#1448) countApiKeyInByomConfig detected apiKey references in agent-generated BYOM configs. The SKILL.md Step 3 only had a brief table mention of bearerToken via DefaultAzureCredential, which was insufficient to steer the agent away from API key patterns. Add explicit BYOM auth constraint to Step 3 and Rules section: MUST use DefaultAzureCredential/ManagedIdentityCredential, NEVER use apiKey or AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY env vars. Bump version to 1.0.4. Update trigger test snapshots for new 'identity' keyword from ManagedIdentityCredential.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses two failing azure-hosted-copilot-sdk integration tests by reducing runtime for a slow greenfield invocation-rate test and tightening BYOM auth guidance to prevent API-key-style configuration leaks.
Changes:
- Reduced the invocation-rate run count for the greenfield-explicit scenario to avoid exceeding the 20-minute Jest timeout.
- Updated
azure-hosted-copilot-sdkskill instructions with a mandatory BYOM auth callout (DefaultAzureCredential/ManagedIdentityCredential only; prohibit API key patterns) and bumped the skill version. - Updated trigger keyword snapshots to reflect the newly introduced
identitykeyword.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/azure-hosted-copilot-sdk/integration.test.ts | Uses GREENFIELD_RUNS_PER_PROMPT = 1 for the greenfield-explicit invocation-rate test to prevent timeouts. |
| plugin/skills/azure-hosted-copilot-sdk/SKILL.md | Adds explicit “no API keys” BYOM auth guidance and bumps skill metadata version to 1.0.4. |
| tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap | Updates expected extracted trigger keywords (adds identity). |
kvenkatrajan
approved these changes
Apr 2, 2026
Jon Gallant (jongio)
added a commit
that referenced
this pull request
Apr 3, 2026
PR #1685 tried to fix BYOM apiKey leak by adding 'NEVER use apiKey' warnings. This backfired — mentioning apiKey in skill context primed the model to emit it. Replace with positive-only auth guidance ('ONLY bearerToken') and add apiKey absence checks to waza eval. Changes: - SKILL.md: Remove all apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY mentions, use positive-only auth framing, bump version 1.0.4 -> 1.0.5 - byom-config.yaml: Add apiKey patterns to output_not_contains Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jon Gallant (jongio)
added a commit
that referenced
this pull request
Apr 3, 2026
#1704) PR #1685 tried to fix BYOM apiKey leak by adding 'NEVER use apiKey' warnings. This backfired — mentioning apiKey in skill context primed the model to emit it. Replace with positive-only auth guidance ('ONLY bearerToken') and add apiKey absence checks to waza eval. Changes: - SKILL.md: Remove all apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY mentions, use positive-only auth framing, bump version 1.0.4 -> 1.0.5 - byom-config.yaml: Add apiKey patterns to output_not_contains Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Barbara 4bes (Ba4bes)
pushed a commit
to Ba4bes/GitHub-Copilot-for-Azure
that referenced
this pull request
Apr 24, 2026
…YOM API key leak (microsoft#1447, microsoft#1448) (microsoft#1685) * fix: reduce greenfield-explicit test runs to prevent timeout (microsoft#1447) Greenfield prompts trigger full planning + scaffolding per agent invocation (~6-8 min each). With RUNS_PER_PROMPT=3, the cumulative runtime exceeded the 20-min TEST_TIMEOUT. Introduce GREENFIELD_RUNS_PER_PROMPT=1 for the greenfield-explicit scenario, matching the pattern used by other slow suites (azure-deploy, azure-prepare). One run is sufficient to validate routing correctness for an explicit SDK mention. * fix: strengthen BYOM auth guidance to prevent API key leaks (microsoft#1448) countApiKeyInByomConfig detected apiKey references in agent-generated BYOM configs. The SKILL.md Step 3 only had a brief table mention of bearerToken via DefaultAzureCredential, which was insufficient to steer the agent away from API key patterns. Add explicit BYOM auth constraint to Step 3 and Rules section: MUST use DefaultAzureCredential/ManagedIdentityCredential, NEVER use apiKey or AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY env vars. Bump version to 1.0.4. Update trigger test snapshots for new 'identity' keyword from ManagedIdentityCredential.
Barbara 4bes (Ba4bes)
pushed a commit
to Ba4bes/GitHub-Copilot-for-Azure
that referenced
this pull request
Apr 24, 2026
…soft#1696) (microsoft#1704) PR microsoft#1685 tried to fix BYOM apiKey leak by adding 'NEVER use apiKey' warnings. This backfired — mentioning apiKey in skill context primed the model to emit it. Replace with positive-only auth guidance ('ONLY bearerToken') and add apiKey absence checks to waza eval. Changes: - SKILL.md: Remove all apiKey/AZURE_OPENAI_API_KEY/AZURE_OPENAI_KEY mentions, use positive-only auth framing, bump version 1.0.4 -> 1.0.5 - byom-config.yaml: Add apiKey patterns to output_not_contains Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1447
Fixes #1448
Summary
Two fixes for
azure-hosted-copilot-sdkintegration test failures.Issue #1447: Greenfield test timeout
The greenfield-explicit test ran
measureInvocationRatewith 3 runs. Each greenfield invocation triggers full planning + code scaffolding (~6-8 min), exceeding the 20-minTEST_TIMEOUT.Fix: Introduced
GREENFIELD_RUNS_PER_PROMPT = 1for the greenfield-explicit scenario, matching the pattern used by other slow suites (azure-deploy, azure-prepare). With 1 run and the 0.6 threshold, the test requires 100% success rate (1/1 = 1.0 >= 0.6), appropriate for an explicit SDK mention.Issue #1448: BYOM config API key leak
countApiKeyInByomConfig(rawMetadata)returned 2 instead of expected 0. The SKILL.md lacked explicit prohibition ofapiKeypatterns in BYOM configuration.Fix: Added a prominent BYOM Auth callout in Step 3 requiring
DefaultAzureCredential/ManagedIdentityCredentialand explicitly prohibitingapiKey,AZURE_OPENAI_API_KEY, andAZURE_OPENAI_KEY. Added matching rule in Rules section. Bumped version 1.0.3 -> 1.0.4.Changes
tests/.../integration.test.tsGREENFIELD_RUNS_PER_PROMPT = 1for slow greenfield testplugin/skills/.../SKILL.mdtests/.../__snapshots__/triggers.test.ts.snapidentitykeyword fromManagedIdentityCredentialTesting