Skip to content

eval: add re-host scenario and fixures - #3042

Merged
anchenyi merged 4 commits into
mainfrom
anchenyi/add-re-host-fixures
Aug 11, 2026
Merged

eval: add re-host scenario and fixures#3042
anchenyi merged 4 commits into
mainfrom
anchenyi/add-re-host-fixures

Conversation

@anchenyi

@anchenyi anchenyi commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Add re-host scenario and fixures

Checklist

  • Tests pass locally (cd tests && npm test)
  • Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
  • If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:integration -- <skill> or npm run test:vally -- --skill <skill>)

Related Issues

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the Microsoft Foundry Vally eval suite by adding a new “migration / re-host” scenario and introducing several Python fixture projects (containerized FastAPI apps + a local MCP tool) that the eval environment can copy in as starting points.

Changes:

  • Added a new “Migration - OpenAI Agents SDK to Foundry” stimulus to the Foundry eval suite, including environment fixture injection.
  • Added three self-contained fixture apps (OpenAI Agents SDK, Google ADK, Claude Agent SDK) exposing /health and /chat, each backed by a local get_order_status MCP tool.
  • Added fixture-level ignore files to prevent committing env/venv/cache artifacts and to reduce Docker build context noise.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated no comments.

Show a summary per file
File Description
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/requirements.txt Pins dependencies for the OpenAI Agents SDK fixture service.
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/mcp_server.py Adds a simple FastMCP stdio server exposing get_order_status.
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/Dockerfile Containerizes the OpenAI Agents SDK fixture as a non-root FastAPI/uvicorn service.
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/agent.py Implements the OpenAI Agents SDK-based FastAPI app that uses the MCP tool.
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/.env.example Documents required env vars for running the OpenAI fixture.
evals/azure-skills/microsoft-foundry/fixture/openai-agents-sdk/.dockerignore Excludes local env/venv/cache artifacts from Docker build context.
evals/azure-skills/microsoft-foundry/fixture/google-adk/requirements.txt Pins dependencies for the Google ADK fixture service.
evals/azure-skills/microsoft-foundry/fixture/google-adk/mcp_server.py Adds the shared get_order_status FastMCP stdio server for the Google ADK fixture.
evals/azure-skills/microsoft-foundry/fixture/google-adk/main.py Implements the Google ADK-based FastAPI app with session/runner wiring.
evals/azure-skills/microsoft-foundry/fixture/google-adk/Dockerfile Containerizes the Google ADK fixture as a non-root FastAPI/uvicorn service.
evals/azure-skills/microsoft-foundry/fixture/google-adk/customer_support_agent/agent.py Defines the Google ADK agent + MCP tool wiring used by the FastAPI app.
evals/azure-skills/microsoft-foundry/fixture/google-adk/customer_support_agent/init.py Marks customer_support_agent as a package for imports.
evals/azure-skills/microsoft-foundry/fixture/google-adk/.env.example Documents required env vars for running the Google fixture.
evals/azure-skills/microsoft-foundry/fixture/google-adk/.dockerignore Excludes local env/venv/cache artifacts from Docker build context.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/requirements.txt Pins dependencies for the Claude Agent SDK fixture service.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/mcp_server.py Adds the shared get_order_status FastMCP stdio server for the Claude fixture.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/Dockerfile Containerizes the Claude Agent SDK fixture as a non-root FastAPI/uvicorn service.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/agent.py Implements the Claude Agent SDK-based FastAPI app configured to call the MCP tool.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/.env.example Documents required env vars for running the Claude fixture.
evals/azure-skills/microsoft-foundry/fixture/claude-agent-sdk/.dockerignore Excludes local env/venv/cache artifacts from Docker build context.
evals/azure-skills/microsoft-foundry/fixture/.gitignore Ignores env/venv/cache artifacts across all fixtures.
evals/azure-skills/microsoft-foundry/eval.yaml Adds a new migration/re-host stimulus and updates the golden-path prompt with a model-selection instruction.

@anchenyi
anchenyi merged commit 3cf3437 into main Aug 11, 2026
10 of 11 checks passed
@anchenyi
anchenyi deleted the anchenyi/add-re-host-fixures branch August 11, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants