feat: replace azure-diagnostics messaging connectivity probe with a script - #2932
Merged
Tom Meschter (tmeschter) merged 10 commits intoAug 6, 2026
Conversation
…cript
Adds cross-platform test-messaging-connectivity.{sh,ps1} scripts that probe DNS resolution, HTTPS reachability, and messaging TCP ports (AMQP 5671/5672, HTTPS 443, optional Kafka 9093) for a Service Bus / Event Hubs namespace, emitting one normalized report. Replaces the inline curl/nslookup probe in service-troubleshooting.md with markdown-linked references and examples.
Fixes microsoft#2511
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Adds an output-content stimulus to evals/azure-diagnostics/eval.yaml that presents a 'cannot connect at all' Service Bus namespace scenario and asserts the skill surfaces the test-messaging-connectivity probe script, exercising the new reference added for microsoft#2511. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Tom Meschter (tmeschter)
requested review from
a team,
Rick Winter (RickWinter) and
Sai Koumudi Kaluvakolanu (saikoumudi)
as code owners
July 21, 2026 20:24
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the azure-diagnostics skill’s messaging troubleshooting flow by replacing an inline, fragile connectivity probe (curl/nslookup + manual port checks) with a dedicated, cross-platform script pair and adds an eval stimulus to ensure the skill surfaces the new probe.
Changes:
- Replaced the inline “Quick Connectivity Test” in
service-troubleshooting.mdwith references and examples fortest-messaging-connectivity.{sh,ps1}. - Added
test-messaging-connectivity.shandtest-messaging-connectivity.ps1scripts that emit a normalized DNS/HTTPS/TCP-port reachability report. - Added an
evals/azure-diagnosticsstimulus asserting the skill output mentionstest-messaging-connectivity.
Show a summary per file
| File | Description |
|---|---|
| plugin/skills/azure-diagnostics/troubleshooting/messaging/service-troubleshooting.md | Replaces inline curl/nslookup probe with script-based probe instructions and links |
| plugin/skills/azure-diagnostics/scripts/test-messaging-connectivity.sh | Adds bash connectivity probe script with normalized reporting |
| plugin/skills/azure-diagnostics/scripts/test-messaging-connectivity.ps1 | Adds PowerShell connectivity probe script with normalized reporting |
| evals/azure-diagnostics/eval.yaml | Adds an integration stimulus expecting the skill to surface the new probe script |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Low
- bash --help no longer prints the shebang line (filter out #! lines) - PowerShell HTTPS check drops -SkipHttpErrorCheck (unavailable in Windows PowerShell 5.1) and extracts the HTTP status from the caught exception across both PS editions - service-troubleshooting.md links the scripts as markdown links and states the working directory (skill root) for the example commands - eval stimulus adds an earlyTerminate to cap cost and documents why area:output is intentional Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Remove [Parameter(Mandatory)] from the PowerShell script's Namespace param and replace it with an explicit empty check that exits 2, avoiding interactive prompts in non-interactive scenarios. Remove the global \Continue = 'Stop' so a stray non-terminating error no longer aborts the whole probe; the DNS/HTTPS/TCP blocks already use explicit -ErrorAction Stop inside try/catch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Rework the azure-diagnostics connectivity-probe eval so it verifies the agent *attempts* to run test-messaging-connectivity (a shell tool call), rather than merely mentioning the script name in prose. Uses a tool-call-match early-terminate that fires on the attempt and the built-in tool-calls grader, mirroring the azure-validate pattern. No live namespace is required since we only observe the attempt, not the result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Validated locally against a live agent (5 runs, 93%). Two fixes from the local run: (1) reframe the prompt as a production-incident triage so it reliably routes to azure-diagnostics (5/5) instead of azure-messaging or no skill; (2) early-terminate on tool-call-result rather than tool-call-match, because vally's tool-calls grader only matches completed tool calls — terminating on the call start dropped the script's result before it could be graded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
…cs-messaging-probe-script
Upstream/main moved plugin/ -> plugins/azure-skills/ and evals/<skill>/ -> evals/azure-skills/<skill>/. The merge relocated modified files via rename detection, but the two new probe scripts (pure additions) had to be moved manually. Also update stale plugin/skills/azure-diagnostics path text in service-troubleshooting.md and migrate the connectivity-probe eval to the newly-added shell-command-invoked grader (terminating on the tool-call start). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
The multi-plugin restructure (microsoft#2872) changed the vally skill-loading model: without a requiredSkills tag every plugin skill is loaded and competes for routing, so the behavior probe test dropped to 0/5 (agent hand-rolled its own DNS probe instead of loading azure-diagnostics). Pin azure-diagnostics so its description survives char-budget truncation, matching the sibling-eval convention. Back to 5/5 skill-invocation + shell-command-invoked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
Address PR review: the committed .sh was mode 100644, so the documented ./scripts/test-messaging-connectivity.sh invocation would fail with a permission error on Linux/macOS after checkout. Set the executable bit (100755) and invoke the script through �ash in both bash examples so the quick connectivity test runs regardless of how the file mode survives the build copy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c
JasonYeMSFT (JasonYeMSFT)
approved these changes
Aug 5, 2026
Sai Koumudi Kaluvakolanu (saikoumudi)
approved these changes
Aug 5, 2026
Tom Meschter (tmeschter)
deleted the
tmeschter-diagnostics-messaging-probe-script
branch
August 6, 2026 16:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2511. Replaces the inline messaging namespace connectivity probe in the
azure-diagnosticsskill with a tested, cross-platform script pair.Previously
troubleshooting/messaging/service-troubleshooting.mdinlined a fragilecurl -v+nslookupprobe. This addsscripts/test-messaging-connectivity.{sh,ps1}that run the mechanical probe and emit one normalized report.Changes
plugin/skills/azure-diagnostics/scripts/test-messaging-connectivity.shand.ps1:.servicebus.windows.netappended automatically) plus optional--kafka/-Kafkafor Event Hubs Kafka port 9093.troubleshooting/messaging/service-troubleshooting.md: replace the inline probe with markdown-linked script references, example invocations, and a one-line description. The judgment-based firewall/ports prose stays in markdown.evals/azure-diagnostics/eval.yamlthat presents a ""cannot connect at all"" Service Bus scenario and asserts the skill surfacestest-messaging-connectivity.Validation
--kafka, and usage error. Caught and fixed a bug where the bashnslookupfallback returned the DNS server IP instead of the resolved host.npm run build(version stamping) succeeds; scripts copied tooutput/.npm run referencesandnpm run frontmatterpass on the built skill.npm run vally validate-stimuluspasses.