Enhance FAOS documentation and fix evaluator contract issues

[Luffy Chen (XOEEst)]

This pull request introduces comprehensive updates to the Foundry Agent Optimizer and evaluation workflows, modernizing the eval.yaml contract and associated documentation, and adding new evaluation prompts. The changes standardize dataset and optimizer configuration fields, deprecate legacy keys, and clarify the generation and usage of evaluation files and workflows. These improvements ensure consistency, better integration with current azd commands, and clearer guidance for both generating and applying evaluations and optimizations.

Key changes include:

1. Modernization of eval.yaml Contract and Documentation

• Standardized the eval.yaml schema to use a nested dataset object with local_uri, name, and version fields, and added validation_dataset. Deprecated legacy fields like dataset_file, dataset_reference, and validation_reference, which are now only normalized in memory for backward compatibility. Updated all references and documentation to reflect this new contract. [1] [2] [3] [4] [5] [6]

• Updated optimizer configuration fields: replaced max_iterations with max_candidates and optimization_config.model with optimization_config.model_search_space, clarifying the allowed optimizer models and how to select or verify them. [1] [2]

2. Improved Evaluation and Optimization Workflows

• Recommended using azd ai agent eval generate over legacy init flows for generating evaluation files, with clear instructions and updated workflow documentation to reflect this best practice. [1] [2] [3]

• Clarified the process for applying optimizer candidates and running optimizations, including validation steps, review points, and safe deployment practices.

3. Expanded and Updated Evaluation Prompts

• Added new integration prompts to evals/microsoft-foundry/eval.yaml for generating evals, scaffolding agents, applying optimizer candidates, evaluating deployed agents, analyzing failures, and enabling continuous evaluation monitoring, each with detailed grader configurations.

4. Backward Compatibility and Data Normalization

• Ensured that legacy fields in existing eval.yaml files are still supported in memory for reading but are not written in new files, maintaining backward compatibility while guiding users toward the new schema. [1] [2] [3]

5. Documentation and Guidance Enhancements

• Updated all relevant markdown documentation to provide clearer, step-by-step guidance for users, including new tables mapping fields to their purposes, and explicit instructions for when and how to generate or update evaluation files and workflows. [1] [2] [3] [4]

These changes collectively modernize the evaluation and optimization process for Foundry agents, making workflows more robust, user-friendly, and aligned with current best practices.## Description

Checklist

• Tests pass locally (cd tests && npm test)
• Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
• If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:integration -- <skill> or npm run test:vally -- --skill <skill>)

Related Issues

[Copilot]

Pull request overview

This PR updates the microsoft-foundry skill documentation and eval suite to reflect the modernized eval.yaml contract (nested dataset object, optimizer option renames) and adds new integration prompts covering Agent Optimizer and evaluation workflows.

Changes:

• Updated multiple Foundry skill reference docs to use dataset.local_uri/dataset.name/dataset.version, validation_dataset, and the new optimizer option keys (max_candidates, optimization_config.model_search_space).
• Revised optimization workflow guidance to prefer azd ai agent eval generate and to pass an explicit --optimize-model.
• Added several new routing integration stimuli to evals/microsoft-foundry/eval.yaml for optimizer/evaluation/monitoring scenarios.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
plugin/skills/microsoft-foundry/SKILL.md	Updates local eval intent parsing guidance to the new eval.yaml contract keys.
plugin/skills/microsoft-foundry/references/agent-metadata-contract.md	Updates the eval.yaml → metadata mapping table and notes legacy key normalization behavior.
plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluation-suite-generation.md	Updates eval.yaml parsing steps for suite generation to new dataset/validation fields.
plugin/skills/microsoft-foundry/foundry-agent/observe/references/deploy-and-setup.md	Updates “reuse/refresh cache” rules to reference dataset.local_uri / registered dataset fields.
plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/optimize-workflow.md	Updates optimizer workflow to prefer eval generate and explicit --optimize-model.
plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/eval-yaml.md	Updates the canonical eval.yaml example and field mapping for optimizer/dataset contract changes.
evals/microsoft-foundry/eval.yaml	Adds new routing integration prompts/graders for optimizer and evaluation workflows.