diff --git a/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md new file mode 100644 index 000000000..1527f2ea9 --- /dev/null +++ b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md @@ -0,0 +1,104 @@ +# Tracing Insights API + +Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights. + +## When to Use + +Use this instead of manual KQL queries when you want **automated anomaly detection** across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed. + +**Prerequisites:** +- App Insights connected to the Foundry project (with `gen_ai.evaluation.result` custom events) +- Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough) + +## Endpoint + +``` +POST https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights +``` + +The API is region-agnostic — any regional endpoint can serve requests for any project. For lowest latency, use the same region as the Foundry project (e.g., `eastus2`, `westus2`, `westcentralus`). If the project region is unknown, use `eastus2` as the default. + +**Query parameters:** +| Parameter | Required | Description | +|--------------------|----------|------------------------------------------------------------------------| +| `startDateTimeUtc` | Yes | ISO 8601 start of analysis window | +| `endDateTimeUtc` | Yes | ISO 8601 end of analysis window | +| `agent` | Yes | Agent name (URL-encoded) | +| `projectId` | Yes | ARM resource ID of the Foundry project (URL-encoded — contains slashes)| +| `top` | No | Max insights to return (default 50) | + +**Auth:** `az account get-access-token --resource https://ai.azure.com` + +**Body:** Must send `{}` (empty JSON object) — POST with no body returns 400. + +## Example + +```powershell +$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv +$encodedAgent = [uri]::EscapeDataString("my-agent") +$encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}") + +$uri = "https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50" + +$response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{ + "Authorization" = "Bearer $token" + "Content-Type" = "application/json" +} -Body "{}" +``` + +## Response Structure (v1-beta2) + +Response is grouped by agent version. Each insight includes `relatedSpans` with `operationId` (App Insights trace ID) for querying full trace content. + +```json +{ + "agents": [{ + "agent": "my-agent:1", + "insights": [{ + "id": "anomaly-token-shift-", + "type": "Token", + "severity": "Critical", + "message": "Token usage increased by 137%", + "agentVersion": "1", + "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 }, + "relatedSpans": { + "totalCount": 13, + "spans": [ + { "responseId": "resp_...", "operationId": "", "evaluationRunId": null } + ] + } + }], + "insightCount": 3 + }], + "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1 +} +``` + +## Querying Traces from relatedSpans + +Use `operationId` from `relatedSpans` to fetch full trace content from App Insights: + +```kql +dependencies +| where operation_Id == "" +| where customDimensions has "invoke_agent" +| project input = customDimensions["gen_ai.input.messages"], + output = customDimensions["gen_ai.output.messages"], + tokens = toint(customDimensions["gen_ai.usage.output_tokens"]) +``` + +This returns the user query and agent response for the specific trace flagged by the insight. + +## How Changepoint Detection Works + +The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline. + +- 10+ data points give better signal for changepoint detection +- `confidence` close to 1.0 = statistically significant shift + +## Next Steps + +After receiving insights with `Warning` or `Critical` severity: +1. Use `relatedSpans.operationId` values to query full trace content from App Insights (see KQL above) +2. Present the insights summary to the user with severity, type, evaluator name, and shift magnitude +3. Offer to drill into specific traces for detailed analysis using the [trace analysis skill](../trace.md) diff --git a/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md b/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md index 0c1a8cd9e..c81cd919d 100644 --- a/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md +++ b/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md @@ -32,6 +32,7 @@ USE FOR: analyze agent traces, search agent conversations, find failing traces, | "Show me this conversation" / "Trace detail" | [Conversation Detail](references/conversation-detail.md) | | "Find eval results for response ID" / "eval scores from traces" | [Eval Correlation](references/eval-correlation.md) | | "What KQL do I need?" | [KQL Templates](references/kql-templates.md) | +| "Auto-detect agent issues" / "Get automated insights" / "What's wrong with my agent?" | [Tracing Insights API](references/tracing-insights-api.md) | ## Before Starting — Resolve App Insights Connection