Skip to content

feat(azure-diagnostics): add run-ig script for Inspektor Gadget invocation - #2934

Merged
Tom Meschter (tmeschter) merged 6 commits into
microsoft:mainfrom
tmeschter:tmeschter-azure-diagnostics-ig-script
Aug 7, 2026
Merged

feat(azure-diagnostics): add run-ig script for Inspektor Gadget invocation#2934
Tom Meschter (tmeschter) merged 6 commits into
microsoft:mainfrom
tmeschter:tmeschter-azure-diagnostics-ig-script

Conversation

@tmeschter

Copy link
Copy Markdown
Member

Summary

Replaces the inline Inspektor Gadget (IG) invocation wrapper in the azure-diagnostics skill with a cross-platform script pair, per #2508.

Running an IG trace previously required the agent to reconstruct a long, error-prone kubectl debug ... ig run command every time IG was referenced. That assembly is purely mechanical, so it now lives in a script.

Changes

New scriptsplugin/skills/azure-diagnostics/scripts/:

  • run-ig.sh + run-ig.ps1 — resolve the node from a pod (or take --node), inject the pinned IG image/version (mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0), apply the gadget-type default timeout (snapshot_*/top_* = 5s, trace_*/profile_*/tcpdump = 30s), add the k8s namespace/pod/container filters, and handle the tcpdump pcap-ng variant. A --dry-run flag prints the assembled command; the script emits a Ran gadget X on node Y summary plus the gadget JSON.

Docs updated to reference the script (kept the gadget catalog, symptom→gadget map, filter table, timeout/type reference, and interpretation prose):

  • inspektor-gadget.md (base command, node-resolution, and tcpdump blocks replaced with script usage + examples)
  • pod-failures.md, networking.md, node-issues.md, command-flows.md, aks-troubleshooting.md

Execution model

The script executes the gadget, but the reference instructs the agent to ask the user for permission before running it (privileged kubectl debug --profile=sysadmin pod). --dry-run previews the exact command.

Validation

  • Both scripts pass syntax checks (bash -n, PowerShell parser) and produce matching --dry-run output across pod/node/tcpdump/filter cases; error paths (missing target, --pf on a non-tcpdump gadget) handled.
  • npm run build succeeds (stamps azure-diagnostics, copies scripts to output/).
  • Frontmatter validator passes on stamped output; references validator passes (no escaped links, scripts not orphaned).
  • Token check: inspektor-gadget.md kept under the 2000-token limit; no new token regressions.

Notes

  • No tests/azure-diagnostics unit suite or live AKS cluster exists, and the existing evals/azure-diagnostics suite only checks skill routing (early-terminates before IG runs), so live integration coverage of the script is out of scope here.

Fixes #2508

…ation

Replace the inline kubectl debug ... ig run command assembly in the IG
reference with cross-platform run-ig.sh / run-ig.ps1 scripts. The scripts
resolve the node from a pod, inject the pinned IG image/version, apply the
gadget-type default timeout, add k8s filters, and handle the tcpdump variant.
A --dry-run flag prints the assembled command. Update inspektor-gadget.md and
the caller docs (pod-failures, networking, node-issues, command-flows,
aks-troubleshooting) to reference the script instead of hand-built commands.

Fixes microsoft#2508

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92c8ec36-b76b-4825-896f-02598b79135e

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the azure-diagnostics skill to run Inspektor Gadget (IG) via new cross-platform helper scripts instead of repeatedly reconstructing a long kubectl debug ... ig run command in prose, and updates AKS troubleshooting references to point at the script-based invocation.

Changes:

  • Added run-ig.sh and run-ig.ps1 to assemble and run the pinned IG invocation (node resolution, default timeouts, pod/container scoping, tcpdump variant).
  • Updated the IG reference doc to describe the script usage and options instead of the inline command wrapper.
  • Updated multiple AKS troubleshooting guides/flows to reference the new script-based workflow.
Show a summary per file
File Description
plugin/skills/azure-diagnostics/troubleshooting/aks/references/inspektor-gadget.md Replaces inline IG command wrapper with run-ig script usage, options, and examples.
plugin/skills/azure-diagnostics/troubleshooting/aks/references/command-flows.md Updates the deep diagnostics flow to use run-ig.
plugin/skills/azure-diagnostics/troubleshooting/aks/pod-failures.md Updates pod failure guidance to invoke IG via run-ig.
plugin/skills/azure-diagnostics/troubleshooting/aks/node-issues.md Updates node-issues guidance to invoke IG via run-ig.
plugin/skills/azure-diagnostics/troubleshooting/aks/networking.md Updates networking guidance to invoke IG via run-ig.
plugin/skills/azure-diagnostics/troubleshooting/aks/aks-troubleshooting.md Updates top-level AKS troubleshooting guide wording to point to run-ig.
plugin/skills/azure-diagnostics/scripts/run-ig.sh New bash script implementing IG command assembly and execution.
plugin/skills/azure-diagnostics/scripts/run-ig.ps1 New PowerShell script implementing IG command assembly and execution.

Review details

Comments suppressed due to low confidence (3)

plugin/skills/azure-diagnostics/scripts/run-ig.sh:174

  • The script prints "Ran gadget ..." before executing kubectl debug. If the command fails, the message is inaccurate; it should say "Running..." before execution and/or print "Ran" only after successful completion.
echo "Ran gadget $GADGET on node $NODE (timeout ${TIMEOUT}s)" >&2

if [[ "$GADGET" == "tcpdump" ]] && command -v tcpdump >/dev/null 2>&1; then
    "${FULL_CMD[@]}" | tcpdump -nvr -
else

plugin/skills/azure-diagnostics/scripts/run-ig.ps1:173

  • The script prints "Ran gadget ..." before executing kubectl debug. If the command fails, the message is inaccurate; it should say "Running..." before execution and/or print "Ran" only after successful completion.
Write-Host "Ran gadget $Gadget on node $Node (timeout ${Timeout}s)"

if ($Gadget -eq 'tcpdump' -and (Get-Command tcpdump -ErrorAction SilentlyContinue)) {
    & kubectl @fullArgs | & tcpdump -nvr -
}

plugin/skills/azure-diagnostics/troubleshooting/aks/networking.md:84

  • This refers to run-ig but doesn’t indicate how to invoke the new scripts (scripts/run-ig.sh / scripts/run-ig.ps1). Readers may interpret this as a command on PATH.
Use the [`run-ig` script](references/inspektor-gadget.md) with `--pod <pod-name> --ns <ns>` and `trace_dns`. Key signals: `rcode=3` (NXDOMAIN), `rcode=2` (SERVFAIL), high `latency` values, queries going to unexpected destinations.
  • Files reviewed: 8/8 changed files
  • Comments generated: 8
  • Review effort level: Low

Comment thread plugin/skills/azure-diagnostics/scripts/run-ig.sh
Comment thread plugin/skills/azure-diagnostics/troubleshooting/aks/pod-failures.md Outdated
Comment thread plugin/skills/azure-diagnostics/troubleshooting/aks/node-issues.md Outdated
Comment thread plugin/skills/azure-diagnostics/troubleshooting/aks/references/command-flows.md Outdated
Comment thread plugin/skills/azure-diagnostics/troubleshooting/aks/networking.md Outdated
Comment thread plugin/skills/azure-diagnostics/scripts/run-ig.sh Outdated
Comment thread plugin/skills/azure-diagnostics/scripts/run-ig.ps1 Outdated
Comment thread plugin/skills/azure-diagnostics/scripts/run-ig.ps1 Outdated
Comment thread plugin/skills/azure-diagnostics/scripts/run-ig.ps1 Outdated
- run-ig.ps1: drop mandatory -Gadget param (avoid interactive prompt in
  non-interactive use); validate explicitly and exit with a clear message.
- run-ig.ps1: remove \Continue='Stop' so Write-Error no longer
  terminates; guard pod node-resolution against empty/failed kubectl output.
- Both scripts: only show the '| tcpdump -nvr -' pipe in the displayed command
  when tcpdump is present, so --dry-run matches real behavior; note when raw
  pcap-ng is emitted instead.
- run-ig.sh: make usage() print only the leading comment block (no script code).
- Docs: reference scripts/run-ig.sh (or run-ig.ps1) explicitly instead of a bare
  run-ig; note PowerShell PascalCase parameter names.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92c8ec36-b76b-4825-896f-02598b79135e
Adds an integration stimulus that drives the agent to invoke the run-ig
Inspektor Gadget helper script and early-terminates on the tool-call-match
the instant the script is invoked, so the privileged kubectl debug never
executes. No live cluster required. Grades skill invocation plus a
tool-calls check confirming the script was actually called.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92c8ec36-b76b-4825-896f-02598b79135e
Comment thread evals/azure-skills/azure-diagnostics/eval.yaml Outdated
…ig eval

Address review feedback: switch the run-ig stimulus earlyTerminate from
tool-call-match to tool-call-result. Terminating on the completed tool call
reliably records the invocation for the grader; with no cluster/kubectl in
CI the underlying kubectl debug fails instantly and harmlessly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92c8ec36-b76b-4825-896f-02598b79135e
…gnostics-ig-script

# Conflicts:
#	evals/azure-skills/azure-diagnostics/eval.yaml
@tmeschter
Tom Meschter (tmeschter) merged commit b42a453 into microsoft:main Aug 7, 2026
12 checks passed
@tmeschter
Tom Meschter (tmeschter) deleted the tmeschter-azure-diagnostics-ig-script branch August 7, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace azure-diagnostics skill Inspektor Gadget invocation wrapper with a script

4 participants