Skip to content

Gap-10: Container Apps Observe (B → A) — Observability playbook + Dapr tracing - #1642

Merged
JasonYeMSFT (JasonYeMSFT) merged 10 commits into
mainfrom
pass-equity-gap-10
May 4, 2026
Merged

Gap-10: Container Apps Observe (B → A) — Observability playbook + Dapr tracing#1642
JasonYeMSFT (JasonYeMSFT) merged 10 commits into
mainfrom
pass-equity-gap-10

Conversation

@paulyuk

Copy link
Copy Markdown
Member

Closes #1618 | Parent: #1608

1 file: container-apps.md — Environment Log Analytics setup, system vs app logs, built-in metrics, distributed tracing, Dapr observability, ARG queries, KQL.

Starting assessment — domain experts should review.

Closes #1618 | Parent: #1608
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 1, 2026 17:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an Azure Container Apps observability reference focused on environment Log Analytics configuration, log/metric sources, and end-to-end tracing (including Dapr), plus practical ARG/KQL queries.

Changes:

  • Documented environment-level Log Analytics setup and how to interpret system vs console logs.
  • Summarized built-in metrics and provided Application Insights/OpenTelemetry SDK setup guidance.
  • Added distributed tracing guidance (including Dapr config) and a small ARG + KQL query library.

Comment thread plugin/skills/appinsights-instrumentation/references/container-apps.md Outdated
@github-actions

github-actions Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor
Details# 🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

fatal: path 'plugin/skills/appinsights-instrumentation/references/container-apps.md' exists on disk, but not in 'origin/main'

📊 Token Change Report

Comparing origin/mainHEAD

Summary

Metric Value
📈 Total Change +1,857 tokens (+204%)
Before 911 tokens
After 2,768 tokens
Files Changed 2

Changed Files

File Before After Change
plugin/skills/appinsights-instrumentation/references/container-apps.md - 1,831 +1831
plugin/skills/appinsights-instrumentation/SKILL.md 911 937 +26 (+3%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown

📊 Token Limit Check Report

Checked: 585 files
Exceeded: 90 files

⚠️ Files Exceeding Token Limits

File Tokens Limit Over By
.github/skills/analyze-skill-issues/SKILL.md 2109 500 +1609
.github/skills/analyze-test-run/SKILL.md 2471 500 +1971
.github/skills/file-test-bug/SKILL.md 628 500 +128
.github/skills/sensei/README.md 3531 2000 +1531
.github/skills/sensei/SKILL.md 3026 500 +2526
.github/skills/sensei/references/EXAMPLES.md 3701 2000 +1701
.github/skills/sensei/references/LOOP.md 4181 2000 +2181
.github/skills/sensei/references/SCORING.md 4299 2000 +2299
.github/skills/skill-authoring/SKILL.md 839 500 +339
plugin/skills/airunway-aks-setup/SKILL.md 1025 500 +525
plugin/skills/appinsights-instrumentation/SKILL.md 937 500 +437
plugin/skills/azure-ai/SKILL.md 820 500 +320
plugin/skills/azure-aigateway/SKILL.md 1261 500 +761
plugin/skills/azure-aigateway/references/policies.md 2342 2000 +342
plugin/skills/azure-cloud-migrate/SKILL.md 848 500 +348
plugin/skills/azure-cloud-migrate/references/services/container-apps/cloudrun-deployment-guide.md 2029 2000 +29
plugin/skills/azure-cloud-migrate/references/services/container-apps/deployment-guide.md 2458 2000 +458
plugin/skills/azure-cloud-migrate/references/services/container-apps/fargate-deployment-guide.md 2587 2000 +587
plugin/skills/azure-cloud-migrate/references/services/functions/lambda-to-functions.md 2600 2000 +600
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/javascript.md 2181 2000 +181
plugin/skills/azure-compliance/SKILL.md 1188 500 +688
plugin/skills/azure-compute/SKILL.md 1090 500 +590
plugin/skills/azure-compute/workflows/vm-recommender/vm-recommender.md 2631 2000 +631
plugin/skills/azure-compute/workflows/vm-troubleshooter/vm-troubleshooter.md 2509 2000 +509
plugin/skills/azure-cost/SKILL.md 1980 500 +1480
plugin/skills/azure-deploy/SKILL.md 1645 500 +1145
plugin/skills/azure-deploy/references/pre-deploy-checklist.md 4692 2000 +2692
plugin/skills/azure-deploy/references/recipes/azd/errors.md 4004 2000 +2004
plugin/skills/azure-deploy/references/troubleshooting.md 2038 2000 +38
plugin/skills/azure-diagnostics/SKILL.md 1423 500 +923
plugin/skills/azure-enterprise-infra-planner/SKILL.md 1002 500 +502
plugin/skills/azure-enterprise-infra-planner/references/constraints/compute-apps.md 2022 2000 +22
plugin/skills/azure-hosted-copilot-sdk/SKILL.md 1263 500 +763
plugin/skills/azure-kubernetes/SKILL.md 2606 500 +2106
plugin/skills/azure-kubernetes/azure-kubernetes-automatic-readiness/SKILL.md 3609 500 +3109
plugin/skills/azure-kusto/SKILL.md 2152 500 +1652
plugin/skills/azure-messaging/SKILL.md 821 500 +321
plugin/skills/azure-prepare/SKILL.md 3359 500 +2859
plugin/skills/azure-prepare/references/aspire.md 4617 2000 +2617
plugin/skills/azure-prepare/references/plan-template.md 2617 2000 +617
plugin/skills/azure-prepare/references/recipes/azd/aspire.md 2275 2000 +275
plugin/skills/azure-prepare/references/recipes/azd/terraform.md 3555 2000 +1555
plugin/skills/azure-prepare/references/research.md 2274 2000 +274
plugin/skills/azure-prepare/references/resources-limits-quotas.md 3322 2000 +1322
plugin/skills/azure-prepare/references/security.md 2147 2000 +147
plugin/skills/azure-prepare/references/services/functions/bicep.md 3127 2000 +1127
plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md 2813 2000 +813
plugin/skills/azure-prepare/references/services/functions/terraform.md 3404 2000 +1404
plugin/skills/azure-prepare/references/services/sql-database/bicep.md 2037 2000 +37
plugin/skills/azure-quotas/SKILL.md 2821 500 +2321
plugin/skills/azure-quotas/references/commands.md 2644 2000 +644
plugin/skills/azure-resource-lookup/SKILL.md 1394 500 +894
plugin/skills/azure-resource-visualizer/SKILL.md 2122 500 +1622
plugin/skills/azure-storage/SKILL.md 1228 500 +728
plugin/skills/azure-upgrade/SKILL.md 1249 500 +749
plugin/skills/azure-upgrade/references/languages/java/INSTRUCTION.md 2724 2000 +724
plugin/skills/azure-upgrade/references/languages/java/package-specific/com.microsoft.azure.management.md 2215 2000 +215
plugin/skills/azure-upgrade/references/languages/java/templates/PLAN_TEMPLATE.md 2411 2000 +411
plugin/skills/azure-upgrade/references/languages/java/templates/PROGRESS_TEMPLATE.md 2315 2000 +315
plugin/skills/azure-upgrade/references/languages/java/templates/SUMMARY_TEMPLATE.md 2190 2000 +190
plugin/skills/azure-upgrade/references/services/functions/automation.md 3463 2000 +1463
plugin/skills/azure-upgrade/references/services/functions/consumption-to-flex.md 2773 2000 +773
plugin/skills/azure-validate/SKILL.md 950 500 +450
plugin/skills/entra-agent-id/SKILL.md 4001 500 +3501
plugin/skills/entra-app-registration/SKILL.md 2070 500 +1570
plugin/skills/entra-app-registration/references/api-permissions.md 2545 2000 +545
plugin/skills/entra-app-registration/references/cli-commands.md 2211 2000 +211
plugin/skills/entra-app-registration/references/console-app-example.md 2752 2000 +752
plugin/skills/entra-app-registration/references/oauth-flows.md 2375 2000 +375
plugin/skills/microsoft-foundry/SKILL.md 3840 500 +3340
plugin/skills/microsoft-foundry/foundry-agent/create/create.md 4315 2000 +2315
plugin/skills/microsoft-foundry/foundry-agent/create/references/toolbox.md 2851 2000 +851
plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md 6133 2000 +4133
plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/eval-datasets.md 2494 2000 +494
plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/generate-seed-dataset.md 2088 2000 +88
plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/trace-to-dataset.md 4325 2000 +2325
plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md 2084 2000 +84
plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md 2941 2000 +941
plugin/skills/microsoft-foundry/foundry-agent/observe/references/continuous-eval.md 3860 2000 +1860
plugin/skills/microsoft-foundry/foundry-agent/trace/references/kql-templates.md 2701 2000 +701
plugin/skills/microsoft-foundry/models/deploy-model/SKILL.md 1640 500 +1140
plugin/skills/microsoft-foundry/models/deploy-model/capacity/SKILL.md 1739 500 +1239
plugin/skills/microsoft-foundry/models/deploy-model/customize/SKILL.md 2235 500 +1735
plugin/skills/microsoft-foundry/models/deploy-model/customize/references/customize-workflow.md 3335 2000 +1335
plugin/skills/microsoft-foundry/models/deploy-model/preset/SKILL.md 1226 500 +726
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/preset-workflow.md 5534 2000 +3534
plugin/skills/microsoft-foundry/quota/quota.md 2288 2000 +288
plugin/skills/microsoft-foundry/quota/references/capacity-planning.md 2080 2000 +80
plugin/skills/microsoft-foundry/references/agent-metadata-contract.md 2373 2000 +373
plugin/skills/microsoft-foundry/references/sdk/foundry-sdk-py.md 2162 2000 +162

Consider moving content to references/ subdirectories.


Automated token analysis. See skill authoring guidelines for best practices.

Simon J (simonjj) and others added 2 commits April 6, 2026 12:28
- Fix: LA workspace not required (was 'requires', now 'by default')
- Fix: add missing --logs-workspace-key CLI param
- Fix: metric dimensions now include replica (was missing on 5/6 metrics)
- Fix: rename Java 'Auto-agent JAR' to 'Agent JAR (manual)' (ACA has no auto-instrumentation)
- Fix: Dapr endpoint placeholder clarified as OTel Collector, not App Insights
- Fix: ARG query checks all containers, not just containers[0]
- Fix: remove undocumented ReplicaCount_d column
- Fix: rename misleading 'Request latency' section to 'Console log volume'
- Add: actual request latency KQL using App Insights requests table

All fixes validated against official Microsoft ACA documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reference

- Bump metadata.version 1.0.2 → 1.0.3 (required by Skill Structure check)
- Link references/container-apps.md from SKILL.md (required by Markdown References check)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 6, 2026 19:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comment thread plugin/skills/appinsights-instrumentation/SKILL.md
Paul Yuknewicz (paulyuk) and others added 2 commits April 6, 2026 13:00
- Dapr: clarify tracing spans are conditional on config, not automatic
- ARG query: use mv-expand kind=outer to include apps with no env vars
- Update trigger test snapshot for new 'observability' keyword

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Snapshot header link goo.gl/fbAQLP → jestjs.io/docs/snapshot-testing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 7, 2026 18:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

Comment thread plugin/skills/appinsights-instrumentation/references/container-apps.md Outdated

@jongio Jon Gallant (jongio) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guide covers environment setup, metrics, SDK integration, tracing, and KQL queries well. Previous feedback on table formatting, ARG queries, and Dapr wording was addressed. Four items below.

Comment thread plugin/skills/appinsights-instrumentation/references/container-apps.md Outdated
Comment thread plugin/skills/appinsights-instrumentation/references/container-apps.md Outdated
Simon J (simonjj) added a commit to simonjj/GitHub-Copilot-for-Azure that referenced this pull request Apr 23, 2026
- Clarify --logs-workspace-id expects Customer ID (GUID), not ARM resource ID
- Add note about Azure Monitor destination table names (no _CL suffix)
- Rename 'Request latency by revision' to 'by app instance' (matches query grouping)
- Add az containerapp env dapr-component set command for ACA-native Dapr config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@simonjj

Copy link
Copy Markdown
Contributor

Hey Paul! Great work on the observability guide — the KQL library and Dapr tracing sections are really solid.

I went through it from the ACA side and found a few small things worth tightening up (workspace ID clarification, table name schema note for Azure Monitor vs Log Analytics, a query title mismatch, and the Dapr deployment command). Nothing major — just precision stuff.

I put the fixes in #2024 which targets your branch directly, so you can just merge it in when you get a chance. Tested across 5 models and everything looks good.

Thanks!

container-apps.md:
- Clarify --logs-workspace-id expects Customer ID (GUID), not ARM ID
- Add Azure Monitor vs Log Analytics table name note (_CL suffix)
- Use secrets pattern for App Insights connection string (secretref)
- Clarify Dapr config is applied via CLI/Bicep, not raw YAML
- Fix query title 'by revision' to 'by instance' (cloud_RoleInstance)

Incorporates Simon's 4 fixes from PR #2024 (closed due to fork
divergence) plus additional copilot-reviewer and JonG items.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 24, 2026 15:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread plugin/skills/appinsights-instrumentation/SKILL.md
@paulyuk

Copy link
Copy Markdown
Member Author

🧪 E2E Skill Validation — Container Apps Observability (PR #1642)

Tested the observability skill docs end-to-end against a live Container App with App Insights in eastus2.

Results

Test Skill Section Status Details
Workspace ID as Customer ID (GUID) Environment-Level Log Analytics customerId returned GUID ca946f03-...; --logs-workspace-id accepted it correctly
Log Analytics env creation Environment-Level Log Analytics Environment created with log-analytics destination, provisioningState: Succeeded
App Insights conn string as secret Application Insights SDK Setup az containerapp secret set + secretref:appinsights-conn pattern works — env var correctly references secret
Console log KQL query (_CL) KQL Query Library ContainerAppConsoleLogs_CL returned rows with Log_s, ContainerAppName_s, RevisionName_s columns (_s suffix)
System log KQL query (_CL) KQL Query Library ContainerAppSystemLogs_CL returned scaling/update events with Reason_s, EventSource_s columns
Azure Monitor table name note System Logs vs Application Logs ContainerAppConsoleLogs (no _CL) returned empty for log-analytics destination — confirms table name guidance is correct
ARG query (apps w/o App Insights) ARG Queries ⚠️ kind=outer in mv-expand is not supported in Azure Resource Graph KQL. Query works after removing kind=outer but may miss apps with no env vars at all
Built-in metrics available Built-in Metrics All documented metrics present: Replicas, Requests, UsageNanoCores, WorkingSetBytes, RestartCount, RxBytes, TxBytes — plus additional metrics like GpuUtilizationPercentage, ResponseTime, resiliency metrics
FQDN query tip Application Insights SDK Setup --query properties.configuration.ingress.fqdn returned valid FQDN; app responded HTTP 200

Key Validations

  • Workspace Customer ID (GUID) works with --logs-workspace-id
  • secretref pattern works for App Insights connection string
  • Correct KQL table name identified — _CL suffix and _s column suffixes for Log Analytics destination
  • ARG query identifies apps with App Insights (our test app correctly found with hasAppInsights > 0)
  • Built-in metrics (Replicas, Requests, CPU, Memory, Restarts, Network) all available

🐛 Bug Found: ARG Query Uses Unsupported KQL Syntax

The ARG query in the skill uses mv-expand kind=outer envVar = container.env but Azure Resource Graph's KQL dialect does not support kind=outer for mv-expand. The error:

mvexpand: invalid value of 'kind' parameter. Supported values: [bag, array]

Fix: Remove kind=outer from the query. Note that without it, apps with no env vars at all will be dropped by mv-expand (false negatives). Consider a two-stage query or coalesce pattern to handle null env arrays.

Command output logs

Resource Group:
rg-pr1642-test in eastus2 (subscription ca5ce512-88e1-44b1-97c6-22caf84fb2b0)

Log Analytics Workspace:
law-pr1642-test — Customer ID: ca946f03-0ce2-4939-b200-4f4aa7704f90

Container Apps Environment:
env-pr1642-test — Log destination: log-analytics, domain: purpleriver-8e330f91.eastus2.azurecontainerapps.io

App Insights:
ai-pr1642-test — Connection string starts with InstrumentationKey=c5031888-...

Container App:
pr1642-test-app — Image: mcr.microsoft.com/k8se/quickstart:latest, FQDN: pr1642-test-app.purpleriver-8e330f91.eastus2.azurecontainerapps.io

Console Log KQL Output (sample):
ContainerAppName_s: pr1642-test-app Log_s: 2026/04/24 15:34:24 Listening on :80... RevisionName_s: pr1642-test-app--0000001 Stream_s: stderr

System Log KQL Output (sample):
`
EventSource_s: ContainerAppController
Reason_s: ContainerAppUpdate
Log_s: Updating containerApp: pr1642-test-app

EventSource_s: KEDA
Reason_s: KEDAScalersStarted
Log_s: Scaler external-push is built
`

Built-in Metrics Available:
UsageNanoCores, WorkingSetBytes, TxBytes, RxBytes, Requests, RestartCount, Replicas, GpuUtilizationPercentage, CoresQuotaUsed, TotalCoresQuotaUsed, ResiliencyConnectTimeouts, ResiliencyRequestRetries, ResiliencyRequestTimeouts, ResponseTime, CpuPercentage, MemoryPercentage

ARG Error:
mvexpand: invalid value of 'kind' parameter. Supported values: [bag, array]

Cleanup: az group delete -n rg-pr1642-test --yes --no-wait executed.

ARG KQL dialect doesn't support kind=outer (only Log Analytics does).
Use mv-expand + isnotempty filter instead. Found via E2E testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@jongio Jon Gallant (jongio) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addresses all my prior feedback cleanly - secrets pattern, Dapr YAML clarification, table name note, and title rename all look good. E2E validation results confirm correctness across the board.

One small suggestion on the ARG query below, otherwise this is solid.

Addresses jongio's non-blocking suggestion: clarify that the ARG query
drops Container Apps with no env vars, so users know the audit isn't
exhaustive.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@paulyuk

Copy link
Copy Markdown
Member Author

📌 Follow-up enhancement filed: #2149

After this PR ships, we plan to refactor Container Apps templates to use a dynamic MCP tool pattern (analogous to Manvir Kaur (@manvkaur)'s Functions work in #1949), which would replace ~2,033 lines of hardcoded templates with runtime fetches from Azure-Samples + 3-tier fallback. This is intentionally not blocking this PR — we want to ship the proven, hand-curated templates now and layer the optimization after.

@jongio Jon Gallant (jongio) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All prior feedback addressed - workspace ID clarification, secrets pattern for connection strings, Log Analytics vs Azure Monitor table naming, Dapr config deployment note, query title corrections, and ARG query limitation docs. E2E validated. Content is solid and actionable. Approving.

@JasonYeMSFT
JasonYeMSFT (JasonYeMSFT) merged commit ecb1cd9 into main May 4, 2026
12 checks passed
@JasonYeMSFT
JasonYeMSFT (JasonYeMSFT) deleted the pass-equity-gap-10 branch May 4, 2026 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gap-10: Container Apps Observe (B → A) — Observability playbook, ARG queries, Dapr tracing

5 participants