Clarify FAOS optimization guidance and custom evaluator contracts

[Luffy Chen (XOEEst)]

This pull request updates documentation and tests for the Microsoft Foundry Agent to clarify and enforce the correct usage of custom evaluator output contracts, improve guidance on environment variable handling, and standardize the structure of the local optimization package. The most important changes are grouped below:

Evaluator Output Contract Enforcement:

• Updated documentation in deploy.md, observe.md, and deploy-and-setup.md to require that custom evaluator prompts use only the runtime-enforced result and reason JSON fields, removing or rewriting any user-provided output-format instructions that conflict (e.g., score/reasoning schemas or duplicate OUTPUT FORMAT blocks). [1] [2] [3] [4]
• Added a unit test in observe/unit.test.ts to verify that documentation enforces the result/reason output contract and does not allow conflicting schemas.

Optimization Package Structure and Environment Variables:

• Standardized the canonical local agent_optimization package to a split-file structure (__init__.py, _config.py, _resolver.py) and documented the public API and responsibilities for each file in both faos-optimize.md and python-patterns.md. [1] [2]
• Clarified that FAOS_OPTIMIZATION_* environment variable aliases should not be generated unless explicitly requested; only AGENT_OPTIMIZATION_* and OPTIMIZATION_CONFIG are part of the base contract.
• Updated guidance to avoid placing platform-reserved AGENT_* variables in user-authored agent.yaml files, especially for hosted agent vNext, and to document rather than inject optional optimization variables by default. [1] [2]

Formatting and Table Consistency:

• Fixed markdown table formatting for improved readability in faos-optimize.md and python-patterns.md. [1] [2] [3]

These changes help ensure that custom evaluators are compatible with the enforced output schema, reduce configuration errors in deployment, and make the optimization package easier to maintain and extend.This pull request clarifies and enforces the output contract for custom evaluator prompts in the Microsoft Foundry Agent documentation and tests. It ensures that all custom evaluator prompts conform to the runtime-enforced JSON schema (result and reason) and removes or rewrites any conflicting user-provided output instructions. The changes also update related documentation to guide users on proper evaluator prompt formatting and add unit tests to verify these requirements.

Evaluator Prompt Output Contract Enforcement:

• Updated documentation in observe.md, deploy.md, and deploy-and-setup.md to require that custom evaluator prompts use only the runtime-enforced output schema: result and reason. All conflicting user instructions (e.g., score/reasoning, duplicate output blocks, or alternate JSON schemas) must be removed or rewritten before creating evaluators. [1] [2] [3]
• Added explicit callouts and tips in documentation to warn users against including extra output-format blocks or alternate schemas in promptText for custom evaluators.

Testing and Validation:

• Added a unit test in observe/unit.test.ts to verify that the documentation and reference files require the custom evaluator output contract and prohibit conflicting schemas.

These changes help prevent schema mismatches and ensure consistent evaluator behavior across the Foundry Agent workflow.## Description

Checklist

• Tests pass locally (cd tests && npm test)
• If modifying skill descriptions: verified routing correctness with integration tests (npm run test:skills:integration -- <skill>)
• If modifying skill USE FOR / DO NOT USE FOR / PREFER OVER clauses: confirmed no routing regressions for competing skills

Related Issues

[Copilot]

Pull request overview

This PR clarifies Microsoft Foundry Agent documentation around (1) the runtime-enforced custom evaluator prompt output contract (result + reason) and (2) FAOS local optimization package structure and environment variable guidance, and adds a unit test intended to enforce the new evaluator-contract guidance.

Changes:

• Update observe/deploy documentation to require custom evaluator prompts to align with the runtime-enforced result/reason JSON contract and to remove conflicting user-provided output schemas (e.g., score/reasoning or duplicate output blocks).
• Standardize FAOS optimization guidance for a split-file local agent_optimization package and tighten guidance around optimization-related environment variables (including avoiding platform-reserved AGENT_* vars in hosted vNext payloads).
• Add a unit test asserting the presence of the new evaluator output-contract guidance in docs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/microsoft-foundry/foundry-agent/observe/unit.test.ts	Adds a unit test intended to enforce documentation requirements for the result/reason custom evaluator output contract.
plugin/skills/microsoft-foundry/foundry-agent/observe/references/deploy-and-setup.md	Adds explicit guidance to remove/rewrite conflicting output-format instructions before creating custom evaluators.
plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md	Adds a behavioral rule + warning callout documenting the enforced result/reason contract and prohibiting conflicting schemas.
plugin/skills/microsoft-foundry/foundry-agent/faos-optimize/references/python-patterns.md	Updates environment variable guidance and documents the canonical split-file agent_optimization package structure.
plugin/skills/microsoft-foundry/foundry-agent/faos-optimize/faos-optimize.md	Updates FAOS optimize workflow docs with canonical package structure and env var guidance; fixes table formatting.
plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md	Adds a deploy workflow callout reinforcing removal of conflicting evaluator output-format instructions.