From 5c212832f0c79a91eabc70d49935f83bb6b83f25 Mon Sep 17 00:00:00 2001 From: Ilya Matiach Date: Fri, 8 May 2026 10:56:39 -0400 Subject: [PATCH 1/2] feat: add Tracing Insights API and FAOS optimization skill references Add skill reference files for the automated insights-to-optimize loop: - tracing-insights-api.md: Tracing Insights API for anomaly detection - faos-optimization.md: FAOS prompt optimization service - insights-to-optimize.md: End-to-end insights-to-optimize workflow Update entry points in trace.md, observe.md, and SKILL.md with new routing keywords (FAOS, tracing insights, quality regression, anomaly detection). Add preview endpoints callout. Use placeholders instead of hardcoded subscription values. Update trigger test snapshots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- plugin/skills/microsoft-foundry/SKILL.md | 6 +- .../foundry-agent/observe/observe.md | 2 + .../observe/references/faos-optimization.md | 93 +++++++++++++++++++ .../references/insights-to-optimize.md | 81 ++++++++++++++++ .../trace/references/tracing-insights-api.md | 85 +++++++++++++++++ .../foundry-agent/trace/trace.md | 1 + .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../trace/__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- .../__snapshots__/triggers.test.ts.snap | 20 +++- 19 files changed, 486 insertions(+), 42 deletions(-) create mode 100644 plugin/skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md create mode 100644 plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md create mode 100644 plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md diff --git a/plugin/skills/microsoft-foundry/SKILL.md b/plugin/skills/microsoft-foundry/SKILL.md index 50b1f1430..f924758d7 100644 --- a/plugin/skills/microsoft-foundry/SKILL.md +++ b/plugin/skills/microsoft-foundry/SKILL.md @@ -1,6 +1,6 @@ --- name: microsoft-foundry -description: "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare)." +description: "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare)." license: MIT metadata: author: Microsoft @@ -25,8 +25,8 @@ This skill includes specialized sub-skills for specific workflows. **Use these i |-----------|-------------|-----------| | **deploy** | Containerize, build, push to ACR, create/update/clone agent deployments | [deploy](foundry-agent/deploy/deploy.md) | | **invoke** | Send messages to an agent, single or multi-turn conversations | [invoke](foundry-agent/invoke/invoke.md) | -| **observe** | Evaluate agent quality, run batch evals, analyze failures, optimize prompts, improve agent instructions, compare versions, set up CI/CD monitoring, and enable continuous production evaluation | [observe](foundry-agent/observe/observe.md) | -| **trace** | Query traces, analyze latency/failures, correlate eval results to specific responses via App Insights `customEvents` | [trace](foundry-agent/trace/trace.md) | +| **observe** | Evaluate agent quality, run batch evals, analyze failures, optimize prompts (via `prompt_optimize` or FAOS), run insights-to-optimize loops, improve agent instructions, compare versions, set up CI/CD monitoring, and enable continuous production evaluation | [observe](foundry-agent/observe/observe.md) | +| **trace** | Query traces, analyze latency/failures, correlate eval results, auto-detect quality regressions via Tracing Insights API | [trace](foundry-agent/trace/trace.md) | | **troubleshoot** | View hosted agent logs, query telemetry, diagnose failures | [troubleshoot](foundry-agent/troubleshoot/troubleshoot.md) | | **create** | Create new hosted agent applications. Supports Microsoft Agent Framework, LangGraph, or custom frameworks in Python or C#, across `responses` or `invocations` protocols. | [create](foundry-agent/create/create.md) | | **faos-optimize** | Convert existing Python agent code to a FAOS (Foundry Agent Optimization Service) optimization-ready version by wiring evaluator-targeted instructions/model/temperature knobs, then stop for review before deployment. | [faos-optimize](foundry-agent/faos-optimize/faos-optimize.md) | diff --git a/plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md b/plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md index 3a1261a67..fa14b2bf7 100644 --- a/plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md +++ b/plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md @@ -29,6 +29,8 @@ USE FOR: evaluate my agent, run an eval, test my agent, check agent quality, run | "Compare agent versions" | [Step 5: Compare](references/compare-iterate.md) | | "Set up CI/CD evals" | [Step 6: CI/CD & Monitoring](references/cicd-monitoring.md) | | "Enable continuous monitoring" / "Set up production monitoring" / "Evaluation results dropping" | [Continuous Eval](references/continuous-eval.md) | +| "Run FAOS optimization" / "Auto-optimize agent" / "Use FAOS" | [FAOS Optimization](references/faos-optimization.md) (prompt agents only) | +| "Auto-detect and fix issues" / "Insights to optimize" / "End-to-end quality loop" | [Insights-to-Optimize Loop](references/insights-to-optimize.md) (prompt agents only) | > ⚠️ **Important:** Before running any evaluation (Step 2), always resolve the selected agent root, metadata file, and environment, then inspect that metadata file plus `.foundry/evaluators/` and `.foundry/datasets/` in that root only. If the cache is missing, stale, or the user wants to refresh it, route through [Step 1: Auto-Setup](references/deploy-and-setup.md) first — even if the user only asked to "evaluate." Do **not** merge `.foundry` cache or source context from sibling agent folders or sibling metadata files. diff --git a/plugin/skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md new file mode 100644 index 000000000..3ab0d1787 --- /dev/null +++ b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md @@ -0,0 +1,93 @@ +# FAOS Optimization (Foundry Agent Optimization Service) + +Automatically optimize agent instructions through an iterative RUN → EVAL → REFLECT loop. FAOS rewrites the agent's system prompt to fix quality regressions detected by evaluators. + +## Scope + +**Prompt agents only.** FAOS reads and rewrites agent instructions via the Foundry Agents API (`POST /agents/{name}/versions`). Hosted agents are not supported in this workflow. + +## Endpoint + +``` +POST https://agents-optimization.westus2.hyena.infra.ai.azure.com/agents-optimization/v1.0/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.MachineLearningServices/workspaces/{ws}/optimize +``` + +> **Preview:** The FAOS endpoint (`westus2.hyena.infra.ai.azure.com`) is preview/canary infrastructure subject to change. When the production URL becomes available, update the base URL accordingly. +> +> **⚠️ Workspace Requirement:** The `{sub}`, `{rg}`, `{ws}` must reference a **registered** `MachineLearningServices/workspaces` resource. CognitiveServices accounts will not work (hardcoded controller route). Unregistered workspaces return 404. To find a valid workspace, run `az ml workspace list --query "[].{name:name, rg:resource_group}" -o table` and use one from your subscription. +> +> Your actual agent project goes in `foundryProjectUrl` in the request body (any region/subscription). + +**Auth:** `az account get-access-token --resource https://ai.azure.com` + +## Request Body + +```json +{ + "agent": { + "foundryProjectUrl": "https://.services.ai.azure.com/api/projects/", + "agentName": "", + "model": "" + }, + "dataset": [ + { + "name": "test_scenario_1", + "prompt": "Representative query that exercises the problem area", + "criteria": [ + { "name": "task_adherence", "instruction": "Describe what correct behavior looks like" } + ] + } + ], + "evaluators": ["task_adherence"], + "options": { + "evalModel": "", + "budget": 3, + "maxIterations": 2, + "strategies": ["instruction"] + } +} +``` + +**Strategies:** `instruction` (GEPA-style prompt rewrite), `skill` (failure-driven), `model` (model-swap). + +## Polling + +POST returns `{"operationId": "opt_xxx", "status": "pending"}`. Poll until complete: + +```powershell +$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv +$base = "https://agents-optimization.westus2.hyena.infra.ai.azure.com/agents-optimization/v1.0/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.MachineLearningServices/workspaces/{ws}" + +# Capture operationId from the initial POST response +$opId = $startResponse.operationId + +do { + Start-Sleep -Seconds 15 + $result = Invoke-RestMethod -Uri "$base/optimize/$opId" -Headers @{"Authorization"="Bearer $token"} +} while ($result.status -in @("pending", "running")) +``` + +## Response (completed) + +```json +{ + "operationId": "opt_xxx", + "status": "completed", + "baseline": { "avgScore": 0.75, "passRate": 0.667, "avgTokens": 1145 }, + "best": { + "avgScore": 0.75, + "passRate": 1.0, + "avgTokens": 555, + "config": { "systemPrompt": "" } + } +} +``` + +## Apply Optimized Instructions + +After FAOS completes, use `agent_update` MCP tool with the optimized prompt from `best.config.systemPrompt`, or PATCH the agent directly via Foundry Agents API (`/agents/?api-version=2025-05-15-preview`). + +## Caveats + +- FAOS may read default instructions ("You are a helpful assistant") instead of the agent's actual prompt — verify by checking `baseline` scores. If baseline doesn't match expected behavior, manually provide instructions in the request. +- Auto-versioning (`keepVersions: true`) may not create versions correctly on new Foundry agents — create versions manually after optimization. diff --git a/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md new file mode 100644 index 000000000..87ace8062 --- /dev/null +++ b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md @@ -0,0 +1,81 @@ +# Insights-to-Optimize Loop + +End-to-end workflow: automatically detect agent quality regressions via the Tracing Insights API, then fix them via FAOS optimization, then verify improvement. + +## When to Use + +Use this when you want a **fully automated quality improvement loop** — no manual KQL analysis, no manual prompt rewriting. The flow is: + +1. Tracing Insights API detects anomalies in evaluation scores +2. Critical/Warning insights are converted to FAOS optimization criteria +3. FAOS rewrites agent instructions to address the regressions +4. Verification confirms token reduction and score improvement + +**vs. manual eval loop (observe.md Step 4):** Use observe.md when you have a batch eval with specific failure clusters. Use this loop when you want insights auto-detected from production traces. + +**Scope:** Prompt agents only (not hosted agents). + +## Prerequisites + +- App Insights connected to Foundry project with evaluation data (`gen_ai.evaluation.result` events) +- Agent created via Foundry Agents API (not legacy Assistants API) +- Agent's Responses API working (`POST /openai/responses` returns valid completions) + +## Flow + +1. **Tracing Insights API** (detect anomalies) +2. **Bridge** (convert insights to FAOS dataset) +3. **FAOS Optimize** (rewrite prompt) +4. **Create v2 agent** with optimized instructions +5. **Verify** (compare v1 vs v2 tokens and scores) + +## Step 1: Call Tracing Insights API + +See [Tracing Insights API reference](../../trace/references/tracing-insights-api.md) for full details. + +Filter results to `Warning` and `Critical` severity insights. + +## Step 2: Convert Insights to FAOS Dataset + +For each insight, map to a FAOS dataset item: + +| Insight Signal | FAOS Criteria Instruction | +|---------------|---------------------------| +| TaskAdherence drop | "Agent should follow task instructions precisely and complete all requested items" | +| Intent Resolution drop | "Agent should correctly interpret user intent and ask clarifying questions" | +| Token spike | "Agent should give concise, focused responses without excessive verbosity" | +| Latency spike | "Agent should respond efficiently without unnecessary tool calls" | +| Error rate increase | "Agent should handle edge cases gracefully without errors" | + +Generate 2-3 representative prompts per insight that exercise the problem area. Use the agent's domain context to make prompts realistic. + +## Step 3: Call FAOS + +See [FAOS Optimization reference](./faos-optimization.md) for endpoint details and workspace requirement. + +Construct request body with the converted dataset and use `"strategies": ["instruction"]` to rewrite the system prompt. + +## Step 4: Apply and Verify + +1. Extract `best.config.systemPrompt` from FAOS response +2. Create a new agent (e.g., `-v2`) or update existing agent with optimized instructions +3. Send the same test queries to both v1 and v2 +4. Compare: + - **Completion tokens** (expect 30-70% reduction from better instructions) + - **Pass rate** on task adherence criteria + - **Response quality** (spot-check a few responses) + +## Example Summary Output + +| Metric | v1 | v2 | +|--------|----|----| +| Completion tokens | 3195 | 1008 | +| Pass rate | 66.7% | 100% | +| Reduction | — | 68.5% | + +## Decision Point + +After verification, ask the user: +- **Keep v2** → Update production agent with optimized instructions +- **Keep v1** → Discard (insights may need more data) +- **Iterate** → Run another FAOS pass with adjusted criteria diff --git a/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md new file mode 100644 index 000000000..26e062d22 --- /dev/null +++ b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md @@ -0,0 +1,85 @@ +# Tracing Insights API + +Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights. + +## When to Use + +Use this instead of manual KQL queries when you want **automated anomaly detection** across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed. + +**Prerequisites:** +- App Insights connected to the Foundry project (with `gen_ai.evaluation.result` custom events) +- Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough) + +## Endpoint + +> **Preview:** The Tracing Insights endpoint (`eastus2euap.api.azureml.ms`) is preview/canary infrastructure subject to change. When the production URL becomes available, update the base URL accordingly. + +``` +POST https://eastus2euap.api.azureml.ms/notification/v1-beta1/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights +``` + +**Query parameters:** +| Parameter | Required | Description | +|-----------|----------|-------------| +| `startDateTimeUtc` | Yes | ISO 8601 start of analysis window | +| `endDateTimeUtc` | Yes | ISO 8601 end of analysis window | +| `agent` | Yes | Agent name (URL-encoded) | +| `projectId` | Yes | ARM resource ID of the Foundry project (URL-encoded — contains slashes) | +| `top` | No | Max insights to return (default 50) | + +**Auth:** `az account get-access-token --resource https://ai.azure.com` + +**Body:** Must send `{}` (empty JSON object) — POST with no body returns 400. + +## Example + +```powershell +$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv +$encodedAgent = [uri]::EscapeDataString("my-agent") +$encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}") + +$uri = "https://eastus2euap.api.azureml.ms/notification/v1-beta1/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50" + +$response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{ + "Authorization" = "Bearer $token" + "Content-Type" = "application/json" +} -Body "{}" +``` + +## Response Structure + +```json +{ + "insights": [ + { + "id": "anomaly-evaluator-changepoint-", + "type": "Evaluation", + "severity": "Warning|Critical|Improvement", + "message": "TaskAdherence scores shifted down from avg 1.0 to 0.7", + "metadata": { + "evaluationName": "TaskAdherence", + "meanBefore": 1.0, + "meanAfter": 0.714, + "shift": -0.286, + "confidence": 0.999 + } + } + ], + "totalCount": 3, + "criticalCount": 0, + "warningCount": 2, + "improvementCount": 1 +} +``` + +## How Changepoint Detection Works + +The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline. + +- Order and diversity of queries matters — mix easy and hard scenarios +- 10+ data points give better signal for changepoint detection +- `confidence` close to 1.0 = statistically significant shift + +## Next Steps + +After receiving insights with `Warning` or `Critical` severity, route to [FAOS Optimization](../../observe/references/faos-optimization.md) or the [Insights-to-Optimize loop](../../observe/references/insights-to-optimize.md) to automatically improve the agent. diff --git a/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md b/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md index 0c1a8cd9e..c81cd919d 100644 --- a/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md +++ b/plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md @@ -32,6 +32,7 @@ USE FOR: analyze agent traces, search agent conversations, find failing traces, | "Show me this conversation" / "Trace detail" | [Conversation Detail](references/conversation-detail.md) | | "Find eval results for response ID" / "eval scores from traces" | [Eval Correlation](references/eval-correlation.md) | | "What KQL do I need?" | [KQL Templates](references/kql-templates.md) | +| "Auto-detect agent issues" / "Get automated insights" / "What's wrong with my agent?" | [Tracing Insights API](references/tracing-insights-api.md) | ## Before Starting — Resolve App Insights Connection diff --git a/tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap index 885d27efb..5a3ad10f9 100644 --- a/tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap index d19a25d76..10ac345a0 100644 --- a/tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill description trig "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill description trig "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill description trig "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill description trig "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill keywords match s "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill keywords match s "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill keywords match s "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill keywords match s "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`create - Trigger Tests Trigger Keywords Snapshot skill keywords match s "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap index 7c22f3626..29a1aea9d 100644 --- a/tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill description trig "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill description trig "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill description trig "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill description trig "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match s "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match s "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match s "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match s "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match s "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/eval-datasets/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/eval-datasets/__snapshots__/triggers.test.ts.snap index e1715dfc8..4c9caaf44 100644 --- a/tests/microsoft-foundry/foundry-agent/eval-datasets/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/eval-datasets/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill descripti "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill descripti "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill descripti "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill descripti "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill keywords "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill keywords "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill keywords "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill keywords "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`eval-datasets - Trigger Tests Trigger Keywords Snapshot skill keywords "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap index 7645b84bb..77d5c08db 100644 --- a/tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill description trig "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill description trig "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill description trig "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill description trig "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill keywords match s "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill keywords match s "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill keywords match s "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill keywords match s "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`invoke - Trigger Tests Trigger Keywords Snapshot skill keywords match s "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap index 481a95bb9..fb00d3714 100644 --- a/tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill description tri "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill description tri "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill description tri "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill description tri "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill keywords match "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill keywords match "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill keywords match "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill keywords match "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`observe - Trigger Tests Trigger Keywords Snapshot skill keywords match "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap index 59f9bea18..84d2ab0ff 100644 --- a/tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill description trigg "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill description trigg "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill description trigg "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill description trigg "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill keywords match sn "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill keywords match sn "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill keywords match sn "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill keywords match sn "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`trace - Trigger Tests Trigger Keywords Snapshot skill keywords match sn "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap index 92632a3ab..1b018a48c 100644 --- a/tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill descriptio "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill descriptio "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill descriptio "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill descriptio "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill keywords m "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill keywords m "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill keywords m "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill keywords m "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`troubleshoot - Trigger Tests Trigger Keywords Snapshot skill keywords m "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap index dfc211036..1871f3433 100644 --- a/tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill description tr "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill description tr "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill description tr "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill description tr "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill keywords match "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill keywords match "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill keywords match "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill keywords match "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`capacity - Trigger Tests Trigger Keywords Snapshot skill keywords match "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap index 885d27efb..5a3ad10f9 100644 --- a/tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap index 885d27efb..5a3ad10f9 100644 --- a/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap index 885d27efb..5a3ad10f9 100644 --- a/tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill descr "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -83,6 +90,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -101,11 +109,13 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -113,6 +123,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -123,6 +134,7 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -131,17 +143,19 @@ exports[`microsoft-foundry - Trigger Tests Trigger Keywords Snapshot skill keywo "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", diff --git a/tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap b/tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap index 6895aa5b1..a6f269440 100644 --- a/tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap +++ b/tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap @@ -2,11 +2,12 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, agent.yaml, dataset curation from traces. USE FOR: deploy agent to Foundry, hosted agent, create agent, invoke agent, evaluate agent, run batch eval, continuous eval, continuous monitoring, continuous eval status, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, improve agent instructions, optimize system prompt, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", + "description": "Deploy, evaluate, and manage Foundry agents end-to-end: Docker build, ACR push, hosted/prompt agent create, container start, batch eval, continuous eval, prompt optimizer workflows, FAOS optimization, tracing insights, agent.yaml, dataset curation from traces. USE FOR: deploy agent, hosted agent, create agent, invoke agent, evaluate agent, batch eval, continuous eval, optimize prompt, improve prompt, optimize system prompt, improve agent instructions, FAOS, tracing insights, quality regression, anomaly detection, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, create dataset from traces, dataset versioning, eval trending, create AI Services, Cognitive Services, create Foundry resource, provision resource, knowledge index, agent monitoring, customize deployment, onboard, availability. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).", "extractedKeywords": [ "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -25,11 +26,13 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -37,6 +40,7 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -47,6 +51,7 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -55,17 +60,19 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", @@ -82,6 +89,7 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "agent", "agents", "ai", + "anomaly", "assignment", "availability", "azure", @@ -100,11 +108,13 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "dataset", "deploy", "deployment", + "detection", "docker", "end-to-end", "eval", "evaluate", "failure", + "faos", "foundry", "from", "functions", @@ -112,6 +122,7 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "hosted", "improve", "index", + "insights", "instructions", "invoke", "knowledge", @@ -122,6 +133,7 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "monitor", "monitoring", "onboard", + "optimization", "optimize", "optimizer", "permissions", @@ -130,17 +142,19 @@ exports[`microsoft-foundry:resource/create - Trigger Tests Trigger Keywords Snap "prompt", "provision", "push", + "quality", "quota", "rbac", "region", + "regression", "resource", "role", "service", "services", "start", - "status", "system", "traces", + "tracing", "trending", "troubleshoot", "validation", From 3e250c04b8182f7fa937e96783f79eda6799b04a Mon Sep 17 00:00:00 2001 From: Ilya Matiach Date: Fri, 8 May 2026 16:17:09 -0400 Subject: [PATCH 2/2] Update skill files to v1-beta2 API with relatedSpans - tracing-insights-api.md: Update endpoint from v1-beta1 to v1-beta2, document new agents[] response structure, add relatedSpans with operationId for trace querying, add KQL snippet for fetching traces - insights-to-optimize.md: Add Step 2 for extracting traces from relatedSpans operationIds before building FAOS dataset Validated E2E on ilmat-2951: 11 insights, 49 spans with operationIds, 9/12 traces with full content from App Insights. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/insights-to-optimize.md | 21 ++++++- .../trace/references/tracing-insights-api.md | 60 ++++++++++++------- 2 files changed, 56 insertions(+), 25 deletions(-) diff --git a/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md index 87ace8062..b42457082 100644 --- a/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md +++ b/plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md @@ -35,7 +35,22 @@ See [Tracing Insights API reference](../../trace/references/tracing-insights-api Filter results to `Warning` and `Critical` severity insights. -## Step 2: Convert Insights to FAOS Dataset +## Step 2: Extract Traces from relatedSpans + +Each insight in v1-beta2 includes `relatedSpans` with `operationId` values. Query App Insights to get the actual user queries and agent responses: + +```kql +dependencies +| where operation_Id in ("", "") +| where customDimensions has "invoke_agent" +| project query = parse_json(customDimensions["gen_ai.input.messages"]), + response = parse_json(customDimensions["gen_ai.output.messages"]), + tokens = toint(customDimensions["gen_ai.usage.output_tokens"]) +``` + +Use the extracted queries as FAOS dataset prompts. If `relatedSpans` is empty, fall back to manually crafted queries. + +## Step 3: Convert Insights to FAOS Dataset For each insight, map to a FAOS dataset item: @@ -49,13 +64,13 @@ For each insight, map to a FAOS dataset item: Generate 2-3 representative prompts per insight that exercise the problem area. Use the agent's domain context to make prompts realistic. -## Step 3: Call FAOS +## Step 4: Call FAOS See [FAOS Optimization reference](./faos-optimization.md) for endpoint details and workspace requirement. Construct request body with the converted dataset and use `"strategies": ["instruction"]` to rewrite the system prompt. -## Step 4: Apply and Verify +## Step 5: Apply and Verify 1. Extract `best.config.systemPrompt` from FAOS response 2. Create a new agent (e.g., `-v2`) or update existing agent with optimized instructions diff --git a/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md index 26e062d22..10bb1c445 100644 --- a/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md +++ b/plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md @@ -15,7 +15,7 @@ Use this instead of manual KQL queries when you want **automated anomaly detecti > **Preview:** The Tracing Insights endpoint (`eastus2euap.api.azureml.ms`) is preview/canary infrastructure subject to change. When the production URL becomes available, update the base URL accordingly. ``` -POST https://eastus2euap.api.azureml.ms/notification/v1-beta1/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights +POST https://eastus2euap.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights ``` **Query parameters:** @@ -38,7 +38,7 @@ $token = az account get-access-token --resource https://ai.azure.com --query acc $encodedAgent = [uri]::EscapeDataString("my-agent") $encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}") -$uri = "https://eastus2euap.api.azureml.ms/notification/v1-beta1/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50" +$uri = "https://eastus2euap.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50" $response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{ "Authorization" = "Bearer $token" @@ -46,37 +46,53 @@ $response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{ } -Body "{}" ``` -## Response Structure +## Response Structure (v1-beta2) + +Response is grouped by agent version. Each insight includes `relatedSpans` with `operationId` (App Insights trace ID) for querying full trace content. ```json { - "insights": [ - { - "id": "anomaly-evaluator-changepoint-", - "type": "Evaluation", - "severity": "Warning|Critical|Improvement", - "message": "TaskAdherence scores shifted down from avg 1.0 to 0.7", - "metadata": { - "evaluationName": "TaskAdherence", - "meanBefore": 1.0, - "meanAfter": 0.714, - "shift": -0.286, - "confidence": 0.999 + "agents": [{ + "agent": "my-agent:1", + "insights": [{ + "id": "anomaly-token-shift-", + "type": "Token", + "severity": "Critical", + "message": "Token usage increased by 137%", + "agentVersion": "1", + "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 }, + "relatedSpans": { + "totalCount": 13, + "spans": [ + { "responseId": "resp_...", "operationId": "", "evaluationRunId": null } + ] } - } - ], - "totalCount": 3, - "criticalCount": 0, - "warningCount": 2, - "improvementCount": 1 + }], + "insightCount": 3 + }], + "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1 } ``` +## Querying Traces from relatedSpans + +Use `operationId` from `relatedSpans` to fetch full trace content from App Insights: + +```kql +dependencies +| where operation_Id == "" +| where customDimensions has "invoke_agent" +| project input = customDimensions["gen_ai.input.messages"], + output = customDimensions["gen_ai.output.messages"], + tokens = toint(customDimensions["gen_ai.usage.output_tokens"]) +``` + +This returns the user query and agent response — use these to auto-build FAOS optimization datasets from real production traces. + ## How Changepoint Detection Works The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline. -- Order and diversity of queries matters — mix easy and hard scenarios - 10+ data points give better signal for changepoint detection - `confidence` close to 1.0 = statistically significant shift