-
Notifications
You must be signed in to change notification settings - Fork 196
feat: add Tracing Insights API and FAOS optimization skill references #2155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Ilya Matiach (imatiach-msft)
wants to merge
2
commits into
microsoft:main
Choose a base branch
from
imatiach-msft:feat/tracing-insights-faos-skills
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
93 changes: 93 additions & 0 deletions
93
.../skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md
|
imatiach-msft marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # FAOS Optimization (Foundry Agent Optimization Service) | ||
|
|
||
| Automatically optimize agent instructions through an iterative RUN → EVAL → REFLECT loop. FAOS rewrites the agent's system prompt to fix quality regressions detected by evaluators. | ||
|
|
||
| ## Scope | ||
|
|
||
| **Prompt agents only.** FAOS reads and rewrites agent instructions via the Foundry Agents API (`POST /agents/{name}/versions`). Hosted agents are not supported in this workflow. | ||
|
|
||
| ## Endpoint | ||
|
|
||
| ``` | ||
| POST https://agents-optimization.westus2.hyena.infra.ai.azure.com/agents-optimization/v1.0/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.MachineLearningServices/workspaces/{ws}/optimize | ||
| ``` | ||
|
|
||
| > **Preview:** The FAOS endpoint (`westus2.hyena.infra.ai.azure.com`) is preview/canary infrastructure subject to change. When the production URL becomes available, update the base URL accordingly. | ||
| > | ||
| > **⚠️ Workspace Requirement:** The `{sub}`, `{rg}`, `{ws}` must reference a **registered** `MachineLearningServices/workspaces` resource. CognitiveServices accounts will not work (hardcoded controller route). Unregistered workspaces return 404. To find a valid workspace, run `az ml workspace list --query "[].{name:name, rg:resource_group}" -o table` and use one from your subscription. | ||
| > | ||
| > Your actual agent project goes in `foundryProjectUrl` in the request body (any region/subscription). | ||
|
|
||
| **Auth:** `az account get-access-token --resource https://ai.azure.com` | ||
|
|
||
| ## Request Body | ||
|
|
||
| ```json | ||
| { | ||
| "agent": { | ||
| "foundryProjectUrl": "https://<account>.services.ai.azure.com/api/projects/<project>", | ||
| "agentName": "<agent-name>", | ||
| "model": "<model-deployment>" | ||
| }, | ||
| "dataset": [ | ||
| { | ||
| "name": "test_scenario_1", | ||
| "prompt": "Representative query that exercises the problem area", | ||
| "criteria": [ | ||
| { "name": "task_adherence", "instruction": "Describe what correct behavior looks like" } | ||
| ] | ||
| } | ||
| ], | ||
| "evaluators": ["task_adherence"], | ||
| "options": { | ||
| "evalModel": "<model-deployment>", | ||
| "budget": 3, | ||
| "maxIterations": 2, | ||
| "strategies": ["instruction"] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **Strategies:** `instruction` (GEPA-style prompt rewrite), `skill` (failure-driven), `model` (model-swap). | ||
|
|
||
| ## Polling | ||
|
|
||
| POST returns `{"operationId": "opt_xxx", "status": "pending"}`. Poll until complete: | ||
|
|
||
| ```powershell | ||
| $token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv | ||
| $base = "https://agents-optimization.westus2.hyena.infra.ai.azure.com/agents-optimization/v1.0/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.MachineLearningServices/workspaces/{ws}" | ||
|
|
||
| # Capture operationId from the initial POST response | ||
| $opId = $startResponse.operationId | ||
|
|
||
| do { | ||
| Start-Sleep -Seconds 15 | ||
| $result = Invoke-RestMethod -Uri "$base/optimize/$opId" -Headers @{"Authorization"="Bearer $token"} | ||
| } while ($result.status -in @("pending", "running")) | ||
|
imatiach-msft marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| ## Response (completed) | ||
|
|
||
| ```json | ||
| { | ||
| "operationId": "opt_xxx", | ||
| "status": "completed", | ||
| "baseline": { "avgScore": 0.75, "passRate": 0.667, "avgTokens": 1145 }, | ||
| "best": { | ||
| "avgScore": 0.75, | ||
| "passRate": 1.0, | ||
| "avgTokens": 555, | ||
| "config": { "systemPrompt": "<optimized instructions>" } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Apply Optimized Instructions | ||
|
|
||
| After FAOS completes, use `agent_update` MCP tool with the optimized prompt from `best.config.systemPrompt`, or PATCH the agent directly via Foundry Agents API (`/agents/<name>?api-version=2025-05-15-preview`). | ||
|
|
||
| ## Caveats | ||
|
|
||
| - FAOS may read default instructions ("You are a helpful assistant") instead of the agent's actual prompt — verify by checking `baseline` scores. If baseline doesn't match expected behavior, manually provide instructions in the request. | ||
| - Auto-versioning (`keepVersions: true`) may not create versions correctly on new Foundry agents — create versions manually after optimization. | ||
96 changes: 96 additions & 0 deletions
96
...ills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # Insights-to-Optimize Loop | ||
|
|
||
| End-to-end workflow: automatically detect agent quality regressions via the Tracing Insights API, then fix them via FAOS optimization, then verify improvement. | ||
|
|
||
| ## When to Use | ||
|
|
||
| Use this when you want a **fully automated quality improvement loop** — no manual KQL analysis, no manual prompt rewriting. The flow is: | ||
|
|
||
| 1. Tracing Insights API detects anomalies in evaluation scores | ||
| 2. Critical/Warning insights are converted to FAOS optimization criteria | ||
| 3. FAOS rewrites agent instructions to address the regressions | ||
| 4. Verification confirms token reduction and score improvement | ||
|
|
||
| **vs. manual eval loop (observe.md Step 4):** Use observe.md when you have a batch eval with specific failure clusters. Use this loop when you want insights auto-detected from production traces. | ||
|
|
||
| **Scope:** Prompt agents only (not hosted agents). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - App Insights connected to Foundry project with evaluation data (`gen_ai.evaluation.result` events) | ||
| - Agent created via Foundry Agents API (not legacy Assistants API) | ||
| - Agent's Responses API working (`POST /openai/responses` returns valid completions) | ||
|
|
||
| ## Flow | ||
|
|
||
| 1. **Tracing Insights API** (detect anomalies) | ||
| 2. **Bridge** (convert insights to FAOS dataset) | ||
| 3. **FAOS Optimize** (rewrite prompt) | ||
| 4. **Create v2 agent** with optimized instructions | ||
| 5. **Verify** (compare v1 vs v2 tokens and scores) | ||
|
|
||
| ## Step 1: Call Tracing Insights API | ||
|
|
||
| See [Tracing Insights API reference](../../trace/references/tracing-insights-api.md) for full details. | ||
|
|
||
| Filter results to `Warning` and `Critical` severity insights. | ||
|
|
||
| ## Step 2: Extract Traces from relatedSpans | ||
|
|
||
| Each insight in v1-beta2 includes `relatedSpans` with `operationId` values. Query App Insights to get the actual user queries and agent responses: | ||
|
|
||
| ```kql | ||
| dependencies | ||
| | where operation_Id in ("<operationId1>", "<operationId2>") | ||
| | where customDimensions has "invoke_agent" | ||
| | project query = parse_json(customDimensions["gen_ai.input.messages"]), | ||
| response = parse_json(customDimensions["gen_ai.output.messages"]), | ||
| tokens = toint(customDimensions["gen_ai.usage.output_tokens"]) | ||
| ``` | ||
|
|
||
| Use the extracted queries as FAOS dataset prompts. If `relatedSpans` is empty, fall back to manually crafted queries. | ||
|
|
||
| ## Step 3: Convert Insights to FAOS Dataset | ||
|
|
||
| For each insight, map to a FAOS dataset item: | ||
|
|
||
| | Insight Signal | FAOS Criteria Instruction | | ||
| |---------------|---------------------------| | ||
| | TaskAdherence drop | "Agent should follow task instructions precisely and complete all requested items" | | ||
| | Intent Resolution drop | "Agent should correctly interpret user intent and ask clarifying questions" | | ||
| | Token spike | "Agent should give concise, focused responses without excessive verbosity" | | ||
| | Latency spike | "Agent should respond efficiently without unnecessary tool calls" | | ||
| | Error rate increase | "Agent should handle edge cases gracefully without errors" | | ||
|
|
||
| Generate 2-3 representative prompts per insight that exercise the problem area. Use the agent's domain context to make prompts realistic. | ||
|
|
||
| ## Step 4: Call FAOS | ||
|
|
||
| See [FAOS Optimization reference](./faos-optimization.md) for endpoint details and workspace requirement. | ||
|
|
||
| Construct request body with the converted dataset and use `"strategies": ["instruction"]` to rewrite the system prompt. | ||
|
|
||
| ## Step 5: Apply and Verify | ||
|
|
||
| 1. Extract `best.config.systemPrompt` from FAOS response | ||
| 2. Create a new agent (e.g., `<name>-v2`) or update existing agent with optimized instructions | ||
| 3. Send the same test queries to both v1 and v2 | ||
| 4. Compare: | ||
| - **Completion tokens** (expect 30-70% reduction from better instructions) | ||
| - **Pass rate** on task adherence criteria | ||
| - **Response quality** (spot-check a few responses) | ||
|
|
||
| ## Example Summary Output | ||
|
|
||
| | Metric | v1 | v2 | | ||
| |--------|----|----| | ||
| | Completion tokens | 3195 | 1008 | | ||
| | Pass rate | 66.7% | 100% | | ||
| | Reduction | — | 68.5% | | ||
|
|
||
| ## Decision Point | ||
|
|
||
| After verification, ask the user: | ||
| - **Keep v2** → Update production agent with optimized instructions | ||
| - **Keep v1** → Discard (insights may need more data) | ||
| - **Iterate** → Run another FAOS pass with adjusted criteria |
101 changes: 101 additions & 0 deletions
101
...skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Tracing Insights API | ||
|
|
||
| Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights. | ||
|
|
||
| ## When to Use | ||
|
|
||
| Use this instead of manual KQL queries when you want **automated anomaly detection** across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed. | ||
|
imatiach-msft marked this conversation as resolved.
|
||
|
|
||
| **Prerequisites:** | ||
| - App Insights connected to the Foundry project (with `gen_ai.evaluation.result` custom events) | ||
| - Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough) | ||
|
|
||
| ## Endpoint | ||
|
|
||
| > **Preview:** The Tracing Insights endpoint (`eastus2euap.api.azureml.ms`) is preview/canary infrastructure subject to change. When the production URL becomes available, update the base URL accordingly. | ||
|
|
||
| ``` | ||
| POST https://eastus2euap.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights | ||
| ``` | ||
|
|
||
| **Query parameters:** | ||
| | Parameter | Required | Description | | ||
| |-----------|----------|-------------| | ||
| | `startDateTimeUtc` | Yes | ISO 8601 start of analysis window | | ||
| | `endDateTimeUtc` | Yes | ISO 8601 end of analysis window | | ||
| | `agent` | Yes | Agent name (URL-encoded) | | ||
| | `projectId` | Yes | ARM resource ID of the Foundry project (URL-encoded — contains slashes) | | ||
| | `top` | No | Max insights to return (default 50) | | ||
|
|
||
| **Auth:** `az account get-access-token --resource https://ai.azure.com` | ||
|
|
||
| **Body:** Must send `{}` (empty JSON object) — POST with no body returns 400. | ||
|
|
||
| ## Example | ||
|
|
||
| ```powershell | ||
|
imatiach-msft marked this conversation as resolved.
|
||
| $token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv | ||
| $encodedAgent = [uri]::EscapeDataString("my-agent") | ||
| $encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}") | ||
|
|
||
| $uri = "https://eastus2euap.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50" | ||
|
|
||
| $response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{ | ||
| "Authorization" = "Bearer $token" | ||
| "Content-Type" = "application/json" | ||
| } -Body "{}" | ||
| ``` | ||
|
|
||
| ## Response Structure (v1-beta2) | ||
|
|
||
| Response is grouped by agent version. Each insight includes `relatedSpans` with `operationId` (App Insights trace ID) for querying full trace content. | ||
|
|
||
| ```json | ||
| { | ||
| "agents": [{ | ||
| "agent": "my-agent:1", | ||
| "insights": [{ | ||
| "id": "anomaly-token-shift-<hash>", | ||
| "type": "Token", | ||
| "severity": "Critical", | ||
| "message": "Token usage increased by 137%", | ||
| "agentVersion": "1", | ||
| "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 }, | ||
| "relatedSpans": { | ||
| "totalCount": 13, | ||
| "spans": [ | ||
| { "responseId": "resp_...", "operationId": "<trace-id>", "evaluationRunId": null } | ||
| ] | ||
| } | ||
| }], | ||
| "insightCount": 3 | ||
| }], | ||
| "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1 | ||
| } | ||
| ``` | ||
|
|
||
| ## Querying Traces from relatedSpans | ||
|
|
||
| Use `operationId` from `relatedSpans` to fetch full trace content from App Insights: | ||
|
|
||
| ```kql | ||
| dependencies | ||
| | where operation_Id == "<operationId>" | ||
| | where customDimensions has "invoke_agent" | ||
| | project input = customDimensions["gen_ai.input.messages"], | ||
| output = customDimensions["gen_ai.output.messages"], | ||
| tokens = toint(customDimensions["gen_ai.usage.output_tokens"]) | ||
| ``` | ||
|
|
||
| This returns the user query and agent response — use these to auto-build FAOS optimization datasets from real production traces. | ||
|
|
||
| ## How Changepoint Detection Works | ||
|
|
||
| The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline. | ||
|
|
||
| - 10+ data points give better signal for changepoint detection | ||
| - `confidence` close to 1.0 = statistically significant shift | ||
|
|
||
| ## Next Steps | ||
|
|
||
| After receiving insights with `Warning` or `Critical` severity, route to [FAOS Optimization](../../observe/references/faos-optimization.md) or the [Insights-to-Optimize loop](../../observe/references/insights-to-optimize.md) to automatically improve the agent. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.