Skip to content

feat: replace azure-diagnostics AKS pod-failure evidence bundle with a script - #2933

Merged
Tom Meschter (tmeschter) merged 8 commits into
microsoft:mainfrom
tmeschter:tmeschter-aks-evidence-bundle-script
Aug 18, 2026
Merged

feat: replace azure-diagnostics AKS pod-failure evidence bundle with a script#2933
Tom Meschter (tmeschter) merged 8 commits into
microsoft:mainfrom
tmeschter:tmeschter-aks-evidence-bundle-script

Conversation

@tmeschter

Copy link
Copy Markdown
Member

Summary

Closes #2507.

Replaces the repeated read-only AKS pod-failure evidence bundle in the azure-diagnostics skill with a single cross-platform pod-evidence script. Previously the same invariant kubectl sequence (find failing pods → describelogs + --previoustop → resource jsonpath) was spelled out inline and repeated across three docs.

Changes

  • New scripts at plugin/skills/azure-diagnostics/scripts/:
    • pod-evidence.sh (bash) and pod-evidence.ps1 (PowerShell)
    • Gather and digest the bundle per pod into one labeled packet: STATUS, STATE (exit code / reason / last state), EVENTS, current + previous LOGS (tailed), RESOURCES (requests/limits vs live top)
    • Inputs: <pod> -n <namespace>, --all-failing (auto-selects non-Running/Succeeded pods), --tail
    • Read-only, self-describing output. The script only gathers/digests; interpretation stays in the docs' decision tables.
  • Docs updated to reference the script via markdown links with sample invocations and a short description, and to drop the repeated raw command blocks:
    • troubleshooting/aks/pod-failures.md
    • troubleshooting/aks/aks-troubleshooting.md
    • troubleshooting/aks/references/command-flows.md

Validation

  • npm run build (version stamping + script copy into output/) ✅
  • frontmatter (built output) ✅ · references (source + output) ✅
  • tokens check — all three edited docs within limits (net token reduction) ✅
  • Both scripts syntax-checked and smoke-tested (arg validation, --all-failing, single-pod, no-cluster graceful handling); bash/PowerShell output verified at parity
  • .sh stored with LF line endings (per .gitattributes)

Notes

  • Frontmatter version left as 0.0.0-placeholder (NBGV stamps at build).
  • The issue's integration-test step requires the Copilot SDK + a live AKS cluster, which weren't available in this environment. Existing azure-diagnostics evals are routing/invocation-rate only and don't execute the script.

Replace the repeated read-only AKS pod-failure evidence bundle (find failing
pods, describe, logs + --previous, top, resource jsonpath) with a single
cross-platform pod-evidence script (bash + PowerShell) that gathers and digests
the invariant bundle into one labeled packet. Update pod-failures.md,
aks-troubleshooting.md, and command-flows.md to reference the script with
markdown links, sample invocations, and a short description, keeping the
interpretation/decision tables in prose.

Closes microsoft#2507

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the azure-diagnostics AKS pod-failure troubleshooting guidance by replacing repeated inline kubectl evidence-collection command blocks with a single reusable pod-evidence script (bash + PowerShell), and updates docs to reference the script.

Changes:

  • Added pod-evidence.sh and pod-evidence.ps1 to collect a consistent, read-only per-pod evidence digest (status/state/events/logs/resources).
  • Updated AKS troubleshooting docs to link to the scripts and replace repeated command sequences with script invocations.
  • Reduced markdown verbosity by consolidating invariant evidence-gathering steps into scripts.
Show a summary per file
File Description
plugin/skills/azure-diagnostics/troubleshooting/aks/references/command-flows.md Replaces repeated pod describe/logs steps with a link + invocation of the new pod-evidence script.
plugin/skills/azure-diagnostics/troubleshooting/aks/pod-failures.md Introduces an “Evidence Bundle Script” section and rewires symptom sections to reference script output sections.
plugin/skills/azure-diagnostics/troubleshooting/aks/aks-troubleshooting.md Updates the CLI fallback section to call the pod-evidence script rather than individual commands.
plugin/skills/azure-diagnostics/scripts/pod-evidence.sh New bash implementation of the evidence bundle collector/digester.
plugin/skills/azure-diagnostics/scripts/pod-evidence.ps1 New PowerShell implementation of the evidence bundle collector/digester.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 6
  • Review effort level: Low

Comment thread plugin/skills/azure-diagnostics/scripts/pod-evidence.sh
Comment thread plugin/skills/azure-diagnostics/scripts/pod-evidence.sh
Comment thread plugin/skills/azure-diagnostics/scripts/pod-evidence.ps1
Comment thread plugin/skills/azure-diagnostics/troubleshooting/aks/pod-failures.md Outdated
- pod-evidence.sh: replace mapfile with a portable while-read loop (Bash 3.2 / macOS) and validate --tail is a positive integer
- pod-evidence.ps1: use \0 for the STATUS section so failures deterministically print (unable to get pod)
- docs: add relative path to the PowerShell example in pod-failures.md; add PowerShell invocation examples in aks-troubleshooting.md and command-flows.md

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428
…ailures

Adds a response-quality stimulus to evals/azure-diagnostics/eval.yaml that
verifies the agent invokes the read-only pod-evidence.{sh,ps1} script for a
CrashLoopBackOff prompt (issue microsoft#2507). Uses a tool-calls grader to assert the
shell invocation and an earlyTerminate tool-call-match guard to stop the run
right after the attempt; completed grader omitted per early-terminate rule.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Comments suppressed due to low confidence (3)

plugins/azure-skills/skills/azure-diagnostics/scripts/pod-evidence.sh:87

  • With set -euo pipefail, piping kubectl describe ... | ... | head -n 25 can cause sed to exit with SIGPIPE once head exits, which then trips pipefail and aborts the entire script while collecting EVENTS for pods with many events. Avoid head in a pipe here and limit output within a single process (e.g., awk) so the digest continues reliably.
        kubectl describe pod "$pod" -n "$ns" 2>/dev/null | sed -n '/^Events:/,$p' | head -n 25

plugins/azure-skills/skills/azure-diagnostics/scripts/pod-evidence.ps1:45

  • $ErrorActionPreference already defaults to Continue, so setting it explicitly is redundant and goes against the repo’s PowerShell script guidance (avoid adding a redundant $ErrorActionPreference = 'Continue' line). Keeping the explanatory comment is fine; just remove the assignment.
# Best-effort: individual kubectl reads may fail (unreachable cluster, missing
# metrics-server, no previous logs). Keep the default "Continue" so a single failed
# read is suppressed via 2>$null and the digest proceeds instead of aborting.
$ErrorActionPreference = "Continue"

plugins/azure-skills/skills/azure-diagnostics/scripts/pod-evidence.ps1:51

  • -Tail is typed as [int] but isn’t validated as a positive integer (0/negative values will be passed to kubectl logs --tail=... and typically fail). The bash version fails fast with exit code 2; adding equivalent validation here keeps cross-platform behavior consistent.
param(
    [Parameter(Position = 0)][string]$Pod,
    [Alias("n")][string]$Namespace,
    [switch]$AllFailing,
    [int]$Tail = 50
)
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment thread plugins/azure-skills/skills/azure-diagnostics/scripts/pod-evidence.sh Outdated
Address Copilot review on PR microsoft#2933:
- sh: capture --all-failing scan via command substitution and check exit
  status so a kubectl failure exits 1 instead of being misreported as
  "no unhealthy pods found" under set -e.
- sh: read the Events section with single-pass awk instead of piping
  describe into head, avoiding a SIGPIPE abort under set -o pipefail.
- ps1: drop the redundant $ErrorActionPreference = "Continue" (already
  the default); keep the explanatory comment.
- ps1: validate -Tail is a positive integer and exit 2 on bad input,
  matching the bash --tail check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428
@tmeschter

Copy link
Copy Markdown
Member Author

Addressed the remaining lower-confidence suggestions from the latest review in f4f20d5:

  • pod-evidence.sh (EVENTS section): replaced describe ... | head -n 25 with a single-pass awk '/^Events:/{f=1} f && n<25 {print; n++}'. head closing the pipe early could SIGPIPE the upstream kubectl describe and abort the script under set -o pipefail; awk reads all input and self-limits instead.
  • pod-evidence.ps1 ($ErrorActionPreference): dropped the redundant $ErrorActionPreference = "Continue" assignment since Continue is already the default; kept the explanatory comment.
  • pod-evidence.ps1 (-Tail validation): added a positive-integer check that exits 2 on bad input, matching the bash --tail guard for cross-platform parity.

All scripts re-validated (bash -n, PS tokenize, smoke tests for --tail/-Tail bad input → exit 2, --all-failing with no cluster → error + exit 1) and npm run build / references / frontmatter / vally validate-stimulus pass.

Comment thread evals/azure-skills/azure-diagnostics/eval.yaml Outdated
Per PR microsoft#2933 review (JasonYeMSFT): switch the pod-evidence-invoked
stimulus's earlyTerminate from tool-call-match to tool-call-result so
termination keys off the matched call's tool.execution_complete event
(a confirmed invocation with a recorded result) rather than
execution_start, which can be raced by termination before the result is
captured. Update the accompanying comment to match.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428
Resolve upstream AKS diagnostics changes by preserving both the new
aks-baseline script execution eval and this PR's pod-evidence eval and
read-only troubleshooting guidance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428
Resolve the azure-diagnostics eval conflict by preserving the pod-evidence,
messaging connectivity, VM SSH, and AKS baseline stimuli as separate tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdfcb108-93f9-4399-954c-f96110af0428
@tmeschter
Tom Meschter (tmeschter) merged commit 2dc4e04 into microsoft:main Aug 18, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace azure-diagnostics skill AKS pod-failure evidence bundle with a script

4 participants