Optimize FAOS preparation and clarify evaluator documentation - #2482
Merged
Conversation
Address PR microsoft#2174 review comments by expanding FAOS as Foundry Agent Optimization Service on first use and making the Step 8 Python config snippet copy-paste safe. The minimum contract example now imports os and preserves the app's existing model-selection fallback instead of hard-coding MODEL_DEPLOYMENT_NAME unless that is already what the app uses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # plugin/skills/microsoft-foundry/SKILL.md
When creating custom evaluator prompts, treat the runtime-enforced JSON schema as authoritative: result plus reason. Preserve user-provided rubric text, but remove or normalize conflicting output schemas such as score/reasoning or duplicate OUTPUT FORMAT blocks before calling evaluator_catalog_create. Add observe skill test coverage for the promptText guardrail so future edits keep the result/reason contract visible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the 0-to-1 scoringType/minScore/maxScore details from the sample custom evaluator prompt. The important guardrail is the output contract: preserve the rubric, but avoid conflicting output schemas because the runtime enforces result/reason. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore plugin/skills/microsoft-foundry/SKILL.md from upstream/main so this PR no longer carries a line-ending-only change for the top-level skill file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Microsoft Foundry skill documentation and its trigger keyword snapshots to reflect current SDK installation guidance and clearer evaluation terminology, while tightening evaluation-suite naming guidance for consistency with expected service constraints.
Changes:
- Updated the
microsoft-foundryskill description (and corresponding trigger keyword snapshots) to reflect revised capabilities/keywords (notably removing “container start” and adding “agent optimizer”). - Replaced “adaptive evaluator” wording with “rubric-based evaluator” across the observability/evaluation docs.
- Simplified Agent Optimizer setup docs to use the published PyPI package
azure-ai-agentserver-optimizationand added guidance thatsuiteNamemust start with a letter.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/microsoft-foundry/resource/create/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/models/deploy/deploy-model/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/models/deploy/customize-deployment/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/models/deploy/capacity/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/troubleshoot/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/trace/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/observe/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/invoke/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/eval-datasets/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/deploy/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/foundry-agent/create/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/finetuning/snapshots/triggers.test.ts.snap | Updates trigger snapshot to match revised skill description/keywords. |
| tests/microsoft-foundry/snapshots/triggers.test.ts.snap | Updates top-level trigger snapshot to match revised skill description/keywords. |
| plugin/skills/microsoft-foundry/SKILL.md | Updates the skill frontmatter description to align with updated terminology/capabilities and expected routing keywords. |
| plugin/skills/microsoft-foundry/references/agent-metadata-contract.md | Adds explicit suite naming guidance (suiteName must start with a letter). |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluation-suite-generation.md | Renames evaluator terminology and documents suite naming constraints for generated suites. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluate-step.md | Updates evaluator terminology to “rubric-based”. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/deploy-and-setup.md | Documents suite naming constraints during auto-setup. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md | Updates evaluator terminology across the observability workflow description/guardrails. |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/scaffold.md | Switches install guidance from preview wheel to PyPI package. |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/python-patterns.md | Switches install guidance from preview wheel to PyPI package and updates verification checklist. |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/eval-yaml.md | Clarifies/normalizes eval.yaml example structure and adds guidance to avoid including the baseline model as a target candidate. |
Justin Gonzales (jugonzales)
approved these changes
Jun 2, 2026
This was referenced Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates the documentation and test snapshots for the Microsoft Foundry skill, focusing on clarifying terminology, updating SDK installation instructions, and enforcing naming conventions for evaluation suites. The most significant changes are grouped below:
SDK Installation and Usage Updates:
azure-ai-agentserver-optimizationpackage directly via PyPI, removing instructions for using a local preview wheel. This simplifies setup and ensures consistency as the package is now publicly available. [1] [2] [3]Evaluator and Evaluation Suite Terminology:
suiteName) must start with a letter, and provided guidance for prefixing numeric-derived names with an alphabetic label (e.g.,suite-). [1] [2] [3]Skill Description and Trigger Keywords:
SKILL.mdand corresponding test snapshots to reflect new capabilities (e.g., "monitoring", "agent optimizer", "invocations WebSocket") and removed deprecated or redundant terms. [1] [2]Checklist
cd tests && npm test)npm run test:skills:integration -- <skill>)USE FOR/DO NOT USE FOR/PREFER OVERclauses: confirmed no routing regressions for competing skillsRelated Issues