Fix azure cost rate limiting by taylorak · Pull Request #1766 · microsoft/GitHub-Copilot-for-Azure

taylorak (taylorak) · 2026-04-07T23:13:18Z

Description

Addresses three issues with the azure-cost skill across 11 files:

Rate-Limit Retry Handling — AB#37441409

The skill had a 40% integration test failure rate caused by Cost Management API 429 responses. The agent was not properly waiting between requests,
violating the per-scope limit of 4 requests/minute.

Changes:

Added rate-limit threshold tables to cost-query/guardrails.md and cost-forecast/guardrails.md documenting all five tiers (per-user, per-scope,
per-tenant, per-client-type)
Added 429 handling sections instructing the agent to read all three x-ms-ratelimit-microsoft.costmanagement-*-retry-after headers and wait for the
longest value
Updated error-handling.md in both cost-query and cost-forecast to reference all three retry-after headers
Expanded rate-limit guidance in cost-query/workflow.md and cost-optimization/workflow.md
Added rate-limit best practice to SKILL.md
Reduced test RUNS_PER_PROMPT from 5→3 and invocationRateThreshold from
0.8→0.6 to stay within per-scope limits

ClientType Request Header

Added --headers "ClientType=GitHubCopilotForAzure" to all az rest commands in cost-query, cost-forecast, and cost-optimization workflows. This enables
per-client-type rate-limit bucketing (2,000 req/min) and telemetry attribution for requests originating from GitHub Copilot.

Filter Schema Casing — AB#37400748

Fixed filter JSON property names from PascalCase → lowercase to match the official Query API docs (
https://learn.microsoft.com/en-us/rest/api/cost-management/query/usage?view=rest-cost-management-2023-11-01):

And→and, Or→or, Not→not
Dimensions→dimensions, Tags→tags
Name→name, Operator→operator, Values→values

Updated in request-body-schema.md, examples.md, and guardrails.md.

Checklist

Tests pass locally — 441 tests ✅
If modifying skill descriptions: verified routing correctness
If modifying USE FOR/DO NOT USE FOR/PREFER OVER: confirmed no routing regressions
Version bumped in skill frontmatter (1.0.0 →
1.1.0)

Related Issues

Fixes AB#37441409 — azure-cost 40% failure rate due to rate-limit retry handling
Fixes AB#37400748 — Forecast filter schema casing mismatch causing API 400 errors
Related to Integration test failure: azure-cost-optimization – Cost Management response assertion [Assertion mismatch] #1626 — azure-cost-optimization integration test keyword assertion mismatch

Update skill documentation to teach the agent proper 429 retry-after header handling across all three Cost Management rate-limit headers: - x-ms-ratelimit-microsoft.costmanagement-qpu-retry-after - x-ms-ratelimit-microsoft.costmanagement-entity-retry-after - x-ms-ratelimit-microsoft.costmanagement-tenant-retry-after The agent must check whichever headers are present, take the maximum retry-after value, and not retry until that duration has elapsed. Skill doc changes: - Add rate limit thresholds table to cost-query and cost-forecast guardrails - Add 'Handling 429 Responses' section with all-headers guidance - Fix 429 retry rows in error-handling.md to reference all 3 headers - Add per-scope rate limit warnings in workflow files - Add rate-limit best practice to SKILL.md, bump version to 1.1.0 Test changes: - Reduce RUNS_PER_PROMPT from 5 to 3 to lower rate limit pressure - Lower invocationRateThreshold from 0.8 to 0.6 (2/3 must pass) - Add new 429 response-quality test for rate limit handling Resolves AB#37441409 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR updates the azure-cost skill documentation and tests to reduce Cost Management API rate-limit flakiness (429s), ensure requests use a ClientType header for improved throttling bucketing/telemetry, and correct filter schema casing to match the Cost Query API.

Changes:

Document improved 429 handling: read all x-ms-ratelimit-microsoft.costmanagement-*-retry-after headers and wait for the maximum.
Add --headers "ClientType=GitHubCopilotForAzure" to az rest examples and reinforce the requirement in workflows/skill guidance.
Update filter JSON schema/examples from PascalCase to lowercase, and reduce integration test repetitions/thresholds to stay under per-scope limits.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/azure-cost/integration.test.ts	Lowers runs/thresholds to reduce rate-limit-driven integration test flakiness.
plugin/skills/azure-cost/SKILL.md	Bumps skill version and adds best-practice guidance for ClientType + 429 retry headers.
plugin/skills/azure-cost/cost-query/workflow.md	Adds ClientType header to `az rest` examples and expands 429 guidance.
plugin/skills/azure-cost/cost-query/request-body-schema.md	Fixes filter schema casing to lowercase keys per API docs.
plugin/skills/azure-cost/cost-query/guardrails.md	Adds rate-limit threshold table and clarifies 429 handling / header semantics.
plugin/skills/azure-cost/cost-query/examples.md	Updates example filter JSON casing (`tags/name/operator/values`).
plugin/skills/azure-cost/cost-query/error-handling.md	Updates 429 retry strategy to check all retry-after headers.
plugin/skills/azure-cost/cost-optimization/workflow.md	Adds explicit guidance to include ClientType header and respect per-scope throttling.
plugin/skills/azure-cost/cost-forecast/workflow.md	Adds ClientType header to `az rest` example.
plugin/skills/azure-cost/cost-forecast/guardrails.md	Adds rate-limit thresholds and 429 handling guidance for Forecast API.
plugin/skills/azure-cost/cost-forecast/error-handling.md	Updates 429 retry strategy to check all retry-after headers.

Jon Gallant (jongio)

Two things to address:

cost-optimization/azure-aks-anomalies.md (line 12) has an az rest call that's missing the new ClientType header. Since SKILL.md now requires it on all Cost Management API requests, this file should be updated too. (Not in this diff, so noting here.)
The test invocation threshold drop (0.8 to 0.6) weakens the quality gate - see inline comment.

Everything else looks solid - the casing fix is consistent across all cost-query files, the 429 retry guidance is thorough, and the rate-limit threshold tables are a useful addition.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Jon Gallant (jongio)

The three fixes are solid and consistent:

429 retry handling now covers all three retry-after headers across every file that references rate limiting
ClientType header is on all az rest code blocks (including the azure-aks-anomalies.md that was missing before)
Filter schema casing is lowercase everywhere it needs to be

One thing still worth resolving: the threshold math on the test. With RUNS_PER_PROMPT=3 and invocationRateThreshold=0.8, the effective requirement is ceil(3 * 0.8) = 3 - all 3 runs must pass. Before, ceil(5 * 0.8) = 4, so 4/5 had to pass (one failure tolerated). The effective pass rate actually went up from 80% to 100%, which makes the test more brittle, not less.

If the rate-limit fix + ClientType bucketing are expected to eliminate 429 flakiness entirely, keeping 0.8 is fine - it just means zero tolerance. If there's still residual risk, threshold=0.67 would restore the previous one-failure tolerance with 3 runs.

Also: cost-query/workflow.md Key Guardrails table (line 102) still says "Filter AND/OR" - should be lowercase to match the rest of the casing changes. One-line fix.

kvenkatrajan · 2026-04-10T22:04:52Z

casing issue and threshold fixed. approved

* Fix azure-cost 40% failure rate from Cost Management API rate limiting Update skill documentation to teach the agent proper 429 retry-after header handling across all three Cost Management rate-limit headers: - x-ms-ratelimit-microsoft.costmanagement-qpu-retry-after - x-ms-ratelimit-microsoft.costmanagement-entity-retry-after - x-ms-ratelimit-microsoft.costmanagement-tenant-retry-after The agent must check whichever headers are present, take the maximum retry-after value, and not retry until that duration has elapsed. Skill doc changes: - Add rate limit thresholds table to cost-query and cost-forecast guardrails - Add 'Handling 429 Responses' section with all-headers guidance - Fix 429 retry rows in error-handling.md to reference all 3 headers - Add per-scope rate limit warnings in workflow files - Add rate-limit best practice to SKILL.md, bump version to 1.1.0 Test changes: - Reduce RUNS_PER_PROMPT from 5 to 3 to lower rate limit pressure - Lower invocationRateThreshold from 0.8 to 0.6 (2/3 must pass) - Add new 429 response-quality test for rate limit handling Resolves AB#37441409 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix casing for request body of forecast and query * fix PR comments * add headers and upgrade invocation rate threshold * update casing in guardrails --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

taylorak (taylorak) and others added 2 commits April 7, 2026 13:55

fix casing for request body of forecast and query

a31fe53

Copilot AI review requested due to automatic review settings April 7, 2026 23:13

Copilot started reviewing on behalf of taylorak (taylorak) April 7, 2026 23:14 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

fix PR comments

e7331e7

Jon Gallant (jongio) reviewed Apr 8, 2026

View reviewed changes

Comment thread tests/azure-cost/integration.test.ts Outdated

add headers and upgrade invocation rate threshold

6e59921

Copilot AI review requested due to automatic review settings April 10, 2026 17:55

Copilot started reviewing on behalf of taylorak (taylorak) April 10, 2026 17:56 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Comment thread tests/azure-cost/integration.test.ts

Comment thread plugin/skills/azure-cost/cost-query/workflow.md

Jon Gallant (jongio) reviewed Apr 10, 2026

View reviewed changes

update casing in guardrails

47c74fe

kvenkatrajan approved these changes Apr 10, 2026

View reviewed changes

taylorak (taylorak) merged commit c4f93cf into microsoft:main Apr 10, 2026
12 checks passed

Navigation Menu

Uh oh!

Fix azure cost rate limiting - #1766

Fix azure cost rate limiting#1766
taylorak (taylorak) merged 5 commits into
microsoft:mainfrom
taylorak:taylorak/fix-azure-cost-rate-limiting

taylorak (taylorak) commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Uh oh!

kvenkatrajan commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

taylorak (taylorak) commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Choose a reason for hiding this comment

Uh oh!

kvenkatrajan commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants