diff --git a/testing/README.md b/testing/README.md index 81c70827..fca32960 100644 --- a/testing/README.md +++ b/testing/README.md @@ -13,5 +13,6 @@ Test frameworks and samples for Copilot Studio agents. | Folder | Description | |--------|-------------| +| [evaluation/](./evaluation/) | Automated evaluation using the Copilot Studio Evaluation API | | [functional/](./functional/) | Functional testing with pytest and response analysis | | [load/](./load/) | Load testing with JMeter | diff --git a/testing/evaluation/EvalGateADO/README.md b/testing/evaluation/EvalGateADO/README.md new file mode 100644 index 00000000..1778d6cf --- /dev/null +++ b/testing/evaluation/EvalGateADO/README.md @@ -0,0 +1,388 @@ +--- +title: Eval Gate for Azure DevOps +parent: Evaluation +grand_parent: Testing +nav_order: 1 +--- +# Eval Gate for Azure DevOps + +New +{: .label .label-green } + +An Azure DevOps pipeline that uses the [Copilot Studio Evaluation API](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-rest-api) as a PR quality gate. When a developer pushes changes to their feature branch and opens a PR, the pipeline automatically: + +1. Packs the solution from the PR branch +2. Imports it into a shared CI Dev environment +3. Runs evaluations against the draft agent +4. Blocks or allows the merge based on a configurable pass rate threshold +5. Publishes results to the ADO **Tests** tab + +## How It Works + +```mermaid +flowchart LR + A[Edit agent in\nCopilot Studio] --> B[Commit & push\nto feature branch] + B --> C[Open PR] + C --> D[Pipeline imports\nto CI Dev] + D --> E[Run evals\non draft] + E --> F{Pass rate\n>= threshold?} + F -->|Yes| G[Merge allowed] + F -->|No| H[Merge blocked] +``` + +## Pipeline in Action + +When a PR is pushed, the pipeline imports the solution, resolves the bot and test set, and runs the evaluation — all visible in the ADO build logs: + +![Pipeline running eval](./assets/pipeline-run.png) + +## Table of Contents + +- [How It Works](#how-it-works) +- [Components](#components) +- [Prerequisites](#prerequisites) +- [Setup](#setup) + - [Part 1: Power Platform](#part-1-power-platform) — Git integration, CI Dev environment, test set, MCS connection + - [Part 2: Authentication](#part-2-authentication) — App registration, refresh token + - [Part 3: Azure Resources](#part-3-azure-resources) — Key Vault, secrets, ARM service principal + - [Part 4: Azure DevOps](#part-4-azure-devops) — Config, service connection, pipeline, branch policy +- [Running It](#running-it) +- [Local Usage](#local-usage) +- [Known Limitations](#known-limitations) + +## Components + +| File | Description | +|------|-------------| +| `eval-config.json` | Environment IDs, bot schema name, pass threshold | +| `scripts/eval-gate.mjs` | Node.js script: MSAL auth + PPAPI client + JUnit output. Single dependency: `@azure/msal-node` | +| `pipelines/eval-gate.yml` | ADO pipeline with self-hosted (pac CLI) and hosted (Build Tools) options | + +--- + +## Prerequisites + +- **Power Platform**: Dev environment as [Managed Environment](https://learn.microsoft.com/en-us/power-platform/admin/managed-environment-overview), connected to ADO Git via [Dataverse git integration](https://learn.microsoft.com/en-us/power-platform/alm/git-integration/connecting-to-git) +- **Copilot Studio**: Agent with a [test set](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-intro) in the Evaluate tab +- **Azure**: Subscription with Key Vault access +- **Azure DevOps**: Project with Git repo, [Power Platform Build Tools](https://marketplace.visualstudio.com/items?itemName=microsoft-IsvExpTools.PowerPlatform-BuildTools) extension +- **Tooling**: [pac CLI](https://learn.microsoft.com/en-us/power-platform/developer/howto/install-cli-net-tool) (requires .NET 10), Node.js 18+, Azure CLI + +--- + +## Setup + +### Part 1: Power Platform + +#### 1.1 Connect Dev Environment to Git + +1. Power Platform Admin Center: enable your Dev environment as **Managed Environment** +2. Copilot Studio → Solutions → **Connect to Git** +3. Choose **Environment binding** → select your ADO org, project, repo +4. Select your **feature branch** (each developer connects to their own) +5. Set Git folder to `src` → **Connect** + +{: .important } +> You **must** use `src` as the Git folder — the pipeline is configured to pack from this path. If you use a different folder, update the `--folder` argument in `pipelines/eval-gate.yml`. + +{: .tip } +> To commit changes: open a solution → **Source control** (left pane) → review changes → **Commit & push**. + +#### 1.2 Create CI Dev Environment + +A dedicated environment where the pipeline imports and tests solutions. No git binding needed. + +```bash +# Install pac CLI (requires .NET 10 — earlier versions produce a DotnetToolSettings.xml error) +dotnet tool install --global Microsoft.PowerApps.CLI.Tool + +# Create the environment +pac admin create --name "CI - MyAgent" --type Developer --region unitedstates + +# Create a Service Principal for pipeline access +pac admin create-service-principal --environment +``` + +{: .important } +> Save the **Application ID**, **Client Secret**, and **Environment URL** from the output — you'll need all three. The `create-service-principal` command automatically assigns the System Administrator role to the SPN in the target environment. + +#### 1.3 Create a Test Set + +1. Open your agent in Copilot Studio → **Evaluate** tab → **New evaluation** +2. Add 10-20 representative test cases covering key scenarios +3. Commit the solution to git — test sets are part of the solution and travel with it on import + +{: .note } +> The script auto-discovers the test set after import — it uses the first available test set. If your agent has multiple test sets, set `testSetName` in `eval-config.json` to target a specific one by display name. + +#### 1.4 Create MCS Connection (Optional — for Authenticated Eval) + +If your agent uses authenticated actions or knowledge sources (e.g., SharePoint connectors), the eval API needs an `mcsConnectionId` to authenticate during the run. See [Manage user profiles and connections](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-edit#manage-user-profiles-and-connections). + +1. Open [Power Automate](https://make.powerautomate.com) **in the CI Dev environment** +2. Go to **Connections** → create a **Microsoft Copilot Studio** connection +3. Click on the connection → copy the ID from the URL: + ``` + .../connections/shared_microsoftcopilotstudio/{mcsConnectionId}/details + ``` + +{: .warning } +> This connection requires interactive OAuth and cannot be created programmatically. It's a one-time manual step per CI environment. Without it, the pipeline still works but authenticated actions and knowledge sources will return empty or error results during eval. + +--- + +### Part 2: Authentication + +#### 2.1 Create Eval API App Registration + +The evaluation API requires **delegated** (user-context) permissions: + +1. [Azure Portal → App Registrations](https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade) → **New registration** +2. Name it (e.g., "Copilot Studio Eval API") +3. Under **API permissions** → **Add a permission** → **APIs my organization uses**: + - Search **Power Platform API** → delegated: `CopilotStudio.MakerOperations.Read`, `CopilotStudio.MakerOperations.ReadWrite` + - Search **Dynamics CRM** → delegated: `user_impersonation` (for bot ID resolution) +4. Click **Grant admin consent** for the tenant +5. Under **Authentication** → **Add a platform** → **Mobile and desktop applications** → redirect URI: `http://localhost` +6. **Authentication** → **Advanced settings** → **Allow public client flows**: **Yes** +7. Copy the **Application (client) ID** + +#### 2.2 Seed the Refresh Token + +```bash +cd scripts && npm install + +# Start device code flow — follow the browser prompt +node eval-gate.mjs auth --config ../eval-config.json +``` + +The token is printed to stdout. Store it in Key Vault (see part 3): + +```bash +az keyvault secret set \ + --vault-name \ + --name copilot-studio-eval-refresh-token \ + --value "" +``` + +Or pipe directly: + +```bash +node eval-gate.mjs auth --config ../eval-config.json \ + | az keyvault secret set \ + --vault-name \ + --name copilot-studio-eval-refresh-token \ + --value @- +``` + +{: .note } +> The pipeline attempts to rotate the refresh token on each run. If MSAL issues a new token, it is written back to Key Vault automatically. MSAL does not always issue a new token on every call, so the existing token may remain. Re-run the `auth` command before the **90-day** expiry to be safe. + +--- + +### Part 3: Azure Resources + +#### 3.1 Create Key Vault + +```bash +az group create --name rg-copilot-cicd --location eastus + +az keyvault create \ + --name \ + --resource-group rg-copilot-cicd \ + --location eastus \ + --enable-rbac-authorization true + +# Grant yourself Secrets Officer to seed secrets +az role assignment create \ + --role "Key Vault Secrets Officer" \ + --assignee \ + --scope +``` + +#### 3.2 Store Secrets in Key Vault + +The pipeline reads three secrets from Key Vault: + +| Secret Name | Value | Source | +|-------------|-------|--------| +| `copilot-studio-eval-refresh-token` | MSAL refresh token | From `node eval-gate.mjs auth` (step 2.2) | +| `copilot-studio-ci-dev-client-secret` | CI Dev SPN client secret | From `pac admin create-service-principal` (step 1.2) | +| `copilot-studio-eval-mcs-connection-id` | MCS connector connection ID | From Power Automate URL (step 1.4, optional) | + +```bash +az keyvault secret set --vault-name \ + --name copilot-studio-eval-refresh-token --value "" + +az keyvault secret set --vault-name \ + --name copilot-studio-ci-dev-client-secret --value "" + +# Required even if not using authenticated eval — use a placeholder value if skipping step 1.4 +az keyvault secret set --vault-name \ + --name copilot-studio-eval-mcs-connection-id --value "${:-none}" +``` + +{: .warning } +> All three secrets must exist in Key Vault. The pipeline's `AzureKeyVault@2` task fails if any named secret is missing. If you're not using authenticated eval (step 1.4), create the `copilot-studio-eval-mcs-connection-id` secret with the value `none`. + +#### 3.3 Create ARM Service Connection SPN + +Create a Service Principal for the pipeline to access Key Vault: + +```bash +az ad sp create-for-rbac \ + --name "copilot-cicd-pipeline" \ + --role "Key Vault Secrets Officer" \ + --scopes /subscriptions//resourceGroups//providers/Microsoft.KeyVault/vaults/ +``` + +Save the **Application ID** and **Password** for step 4.2. + +--- + +### Part 4: Azure DevOps + +#### 4.1 Configure eval-config.json + +Edit `eval-config.json` in your repo root and fill in the values from previous steps: + +```json +{ + "environmentId": "", + "environmentUrl": "https://.crm.dynamics.com/", + "botSchemaName": "", + "tenantId": "", + "clientId": "", + "passThreshold": 0.8 +} +``` + +Commit this file to your repo — the pipeline reads it at runtime. + +| Field | Description | Where to find it | +|-------|-------------|-----------------| +| `environmentId` | CI Dev environment GUID | Power Platform Admin Center → Environments | +| `environmentUrl` | CI Dev Dataverse URL (trailing slash) | Same page, or `pac env who` | +| `botSchemaName` | Bot schema name from the solution | `src/bots//bot.yml` → `@schemaname` | +| `tenantId` | Entra tenant GUID | Azure Portal → Entra ID → Overview | +| `clientId` | Eval API app registration client ID | From step 2.1 | +| `passThreshold` | Pass rate required (0.0-1.0) | Choose your quality bar | + +{: .note } +> `agentId` and `testSetId` are resolved dynamically at runtime — you don't set them. + +#### 4.2 Create ADO Service Connection + +In ADO: Project Settings → Service connections → New → **Azure Resource Manager** → **Service principal (manual)** + +| Field | Value | +|-------|-------| +| Name | `AzureRM-KeyVault` | +| Subscription ID | Your Azure subscription ID | +| Service Principal ID | App ID from step 3.3 | +| Service Principal Key | Password from step 3.3 | +| Tenant ID | Your Entra tenant ID | + +Enable for all pipelines. + +#### 4.3 Update Pipeline Variables + +Edit `pipelines/eval-gate.yml` and set: + +```yaml +variables: + AZURE_SERVICE_CONNECTION: '' + KEY_VAULT_NAME: '' + CI_DEV_ENV_URL: '' + CI_DEV_APP_ID: '' + CI_DEV_TENANT_ID: '' +``` + +Commit and push `eval-config.json` and `pipelines/eval-gate.yml` to your repo before proceeding. + +#### 4.4 Create the Pipeline + +1. ADO → Pipelines → New pipeline → Azure Repos Git → select your repo +2. Point to `pipelines/eval-gate.yml` +3. Save (don't run yet) +4. Pipeline Settings → set max concurrent runs to **1** (sequential execution) + +#### 4.5 Approve Service Connections (First Run Only) + +The first time the pipeline runs, ADO will prompt you to **Permit** access to the `AzureRM-KeyVault` service connection. Click **Permit** — this is a one-time approval. + +#### 4.6 Configure Branch Policy (Merge Gate) + +1. ADO Repos → Branches → `main` → Branch policies +2. **Build validation** → Add → select the eval-gate pipeline +3. Set to **Required** → Trigger: **Automatic** + +--- + +## Running It + +Once setup is complete, the eval gate runs automatically: + +1. **Make a change** to your agent in Copilot Studio (edit a topic, update instructions, etc.) +2. **Commit & push** from the Solutions → Source control page to your feature branch +3. **Open a PR** targeting `main` in Azure DevOps +4. The **pipeline triggers automatically** on push — it packs the solution, imports it into the CI Dev environment, and runs evaluations against the draft agent +5. **Check the results** in the PR: + - The **Summary** tab shows the pipeline pass/fail status + - The **Tests** tab shows individual test case results with metric breakdowns + - The eval results JSON is available as a **pipeline artifact** +6. If the pass rate meets the threshold, the **merge button is enabled**. Otherwise, the PR is blocked until the agent quality improves. + +Subsequent pushes to the same PR branch re-trigger the pipeline — each push gets a fresh eval run. + +**PR gate passed** — eval meets the threshold, merge is allowed: + +![PR with eval gate passed](./assets/pr-gate-passed.png) + +**PR gate failed** — eval below threshold, merge is blocked: + +![PR with eval gate failed](./assets/pr-gate-failed.png) + +**Tests tab** — individual test case results with pass/fail breakdown: + +![Tests tab showing eval results](./assets/tests-tab.png) + +## Local Usage + +Run evals locally against your own Dev environment: + +```bash +export EVAL_REFRESH_TOKEN="" + +# Verify bot ID resolution +node scripts/eval-gate.mjs resolve-bot --config eval-config.json + +# List test sets +node scripts/eval-gate.mjs list-testsets --config eval-config.json + +# Run eval +node scripts/eval-gate.mjs run \ + --config eval-config.json \ + --run-name "local test" \ + --output results.json \ + --junit-output results.junit.xml + +# Override for a different environment +node scripts/eval-gate.mjs run \ + --config eval-config.json \ + --environment-id "" \ + --agent-id "" \ + --threshold 0.9 \ + --run-name "local test" +``` + +--- + +## Known Limitations + +- **Delegated auth only**: The eval API requires user-context tokens. The pipeline uses a pre-cached refresh token (90-day expiry). There is no app-only authentication path. +- **MCS connection is manual**: The Microsoft Copilot Studio connector connection requires interactive OAuth and cannot be created programmatically. See [step 1.4](#14-create-mcs-connection-optional--for-authenticated-eval) for setup. Without it, authenticated actions and knowledge sources won't work during eval. +- **Single CI environment**: Pipeline runs are serialized (max 1 concurrent). Multiple PRs queue and run one at a time. +- **Self-hosted agent**: The `pac` CLI approach requires a self-hosted agent. The pipeline's `DOTNET_ROOT` path is configured for macOS with Homebrew — adjust it for Linux agents. For Microsoft-hosted agents (Windows/Linux), use the commented-out Power Platform Build Tools section in the pipeline YAML. +- **Errors count as failures**: Evaluation errors (e.g., model timeouts, missing MCS connection) count against the pass threshold. If a run produces unexpectedly low scores, check `eval-results.json` for test cases with `Error` status. +- **Test case names**: The eval API returns test case IDs (GUIDs), not display names. Results in the Tests tab show GUIDs. diff --git a/testing/evaluation/EvalGateADO/assets/pipeline-run.png b/testing/evaluation/EvalGateADO/assets/pipeline-run.png new file mode 100644 index 00000000..d0161ef2 Binary files /dev/null and b/testing/evaluation/EvalGateADO/assets/pipeline-run.png differ diff --git a/testing/evaluation/EvalGateADO/assets/pipeline-setup.png b/testing/evaluation/EvalGateADO/assets/pipeline-setup.png new file mode 100644 index 00000000..1c504785 Binary files /dev/null and b/testing/evaluation/EvalGateADO/assets/pipeline-setup.png differ diff --git a/testing/evaluation/EvalGateADO/assets/pr-gate-failed.png b/testing/evaluation/EvalGateADO/assets/pr-gate-failed.png new file mode 100644 index 00000000..f7eb9f74 Binary files /dev/null and b/testing/evaluation/EvalGateADO/assets/pr-gate-failed.png differ diff --git a/testing/evaluation/EvalGateADO/assets/pr-gate-passed.png b/testing/evaluation/EvalGateADO/assets/pr-gate-passed.png new file mode 100644 index 00000000..189ccd91 Binary files /dev/null and b/testing/evaluation/EvalGateADO/assets/pr-gate-passed.png differ diff --git a/testing/evaluation/EvalGateADO/assets/tests-tab.png b/testing/evaluation/EvalGateADO/assets/tests-tab.png new file mode 100644 index 00000000..954c3c96 Binary files /dev/null and b/testing/evaluation/EvalGateADO/assets/tests-tab.png differ diff --git a/testing/evaluation/EvalGateADO/eval-config.json b/testing/evaluation/EvalGateADO/eval-config.json new file mode 100644 index 00000000..14e5c2c4 --- /dev/null +++ b/testing/evaluation/EvalGateADO/eval-config.json @@ -0,0 +1,8 @@ +{ + "environmentId": "", + "environmentUrl": "https://.crm.dynamics.com/", + "botSchemaName": "", + "tenantId": "", + "clientId": "", + "passThreshold": 0.8 +} diff --git a/testing/evaluation/EvalGateADO/pipelines/eval-gate.yml b/testing/evaluation/EvalGateADO/pipelines/eval-gate.yml new file mode 100644 index 00000000..93760c53 --- /dev/null +++ b/testing/evaluation/EvalGateADO/pipelines/eval-gate.yml @@ -0,0 +1,223 @@ +# Copilot Studio Eval Gate Pipeline +# +# Runs Copilot Studio evaluations on every push to a PR targeting main. +# Used as a branch policy gate — PRs cannot merge unless eval pass rate +# meets the threshold defined in eval-config.json. +# +# Prerequisites: +# 1. Azure Key Vault with secrets (see README for full list) +# 2. ADO Service Connection (ARM) with Key Vault Secrets Officer role +# 3. eval-config.json populated with environment details and botSchemaName +# 4. Test set created in Copilot Studio Evaluate tab (travels with the solution) +# 5. Pipeline set to max 1 concurrent run (sequential execution) +# 6. First run: approve service connection permissions when prompted +# +# Agent options: +# - Self-hosted (macOS/Linux): Uses pac CLI directly (current config) +# - Microsoft-hosted (Windows/Linux): Use PowerPlatform Build Tools tasks instead +# (see commented-out section below) + +trigger: none + +# Run eval on every push to a PR targeting main +pr: + branches: + include: + - main + paths: + include: + - src/* + +# --------------------------------------------------------------------------- +# Option A: Self-hosted agent (uses pac CLI directly) +# --------------------------------------------------------------------------- +pool: 'self-hosted' + +# --------------------------------------------------------------------------- +# Option B: Microsoft-hosted agent (uncomment and use instead of Option A) +# --------------------------------------------------------------------------- +# pool: +# vmImage: 'ubuntu-latest' + +variables: + # Azure Resource Manager service connection for Key Vault access + AZURE_SERVICE_CONNECTION: '' + # Key Vault name storing refresh token, MCS connection ID, and CI Dev SPN secret + KEY_VAULT_NAME: '' + # CI Dev environment details (for pac CLI auth) + CI_DEV_ENV_URL: '' # e.g., https://org12345.crm.dynamics.com/ + CI_DEV_APP_ID: '' + CI_DEV_TENANT_ID: '' + +steps: + - checkout: self + + - task: NodeTool@0 + displayName: 'Install Node.js' + inputs: + versionSpec: '18.x' + + - script: cd $(Build.SourcesDirectory)/scripts && npm install + displayName: 'Install dependencies' + + # Fetch all secrets from Key Vault + - task: AzureKeyVault@2 + displayName: 'Fetch secrets from Key Vault' + inputs: + azureSubscription: '$(AZURE_SERVICE_CONNECTION)' + KeyVaultName: '$(KEY_VAULT_NAME)' + SecretsFilter: 'copilot-studio-eval-refresh-token,copilot-studio-eval-mcs-connection-id,copilot-studio-ci-dev-client-secret' + RunAsPreJob: false + + # --------------------------------------------------------------------------- + # Option A: Self-hosted agent — pack, import, and resolve via pac CLI + # --------------------------------------------------------------------------- + - script: | + set -euo pipefail + set +x # Prevent secrets from appearing in debug/trace logs + + export DOTNET_ROOT="/opt/homebrew/opt/dotnet/libexec" + PAC="$HOME/.dotnet/tools/pac" + + # Auth to CI Dev with SPN (secret via env var, not CLI arg) + $PAC auth create \ + --environment "$(CI_DEV_ENV_URL)" \ + --applicationId "$(CI_DEV_APP_ID)" \ + --clientSecret "$SPN_CLIENT_SECRET" \ + --tenant "$(CI_DEV_TENANT_ID)" + + # Pack unmanaged solution + echo "Packing solution from src/..." + $PAC solution pack \ + --zipfile "$(Build.ArtifactStagingDirectory)/solution_unmanaged.zip" \ + --folder "$(Build.SourcesDirectory)/src" + + # Import into CI Dev + echo "Importing solution into CI Dev..." + $PAC solution import \ + --path "$(Build.ArtifactStagingDirectory)/solution_unmanaged.zip" \ + --environment "$(CI_DEV_ENV_URL)" \ + --async + + # Resolve bot ID via Dataverse OData query using SPN client credentials + BOT_SCHEMA_NAME=$(python3 -c "import json; print(json.load(open('eval-config.json'))['botSchemaName'])") + ENV_URL=$(python3 -c "import json; print(json.load(open('eval-config.json'))['environmentUrl'])") + + echo "Resolving bot ID for: $BOT_SCHEMA_NAME in $ENV_URL" + + TOKEN=$(curl -sf --no-progress-meter -X POST \ + "https://login.microsoftonline.com/${SPN_TENANT_ID}/oauth2/v2.0/token" \ + -d "client_id=${SPN_APP_ID}" \ + -d "client_secret=${SPN_CLIENT_SECRET}" \ + -d "scope=${ENV_URL}.default" \ + -d "grant_type=client_credentials" 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + + BOT_ID=$(curl -sf "${ENV_URL}api/data/v9.2/bots?\$filter=schemaname%20eq%20'${BOT_SCHEMA_NAME}'&\$select=botid" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Accept: application/json" \ + -H "OData-MaxVersion: 4.0" \ + -H "OData-Version: 4.0" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['value'][0]['botid'])") + + echo "Resolved bot ID: $BOT_ID" + echo "##vso[task.setvariable variable=RESOLVED_BOT_ID]$BOT_ID" + displayName: 'Pack, import, and resolve bot ID' + env: + SPN_TENANT_ID: $(CI_DEV_TENANT_ID) + SPN_APP_ID: $(CI_DEV_APP_ID) + SPN_CLIENT_SECRET: $(copilot-studio-ci-dev-client-secret) + + # --------------------------------------------------------------------------- + # Option B: Microsoft-hosted agent — use Power Platform Build Tools instead + # (uncomment this section and comment out Option A above) + # --------------------------------------------------------------------------- + # - task: PowerPlatformToolInstaller@2 + # displayName: 'Install Power Platform CLI' + # + # - task: PowerPlatformPackSolution@2 + # displayName: 'Pack unmanaged solution' + # inputs: + # SolutionSourceFolder: '$(Build.SourcesDirectory)/src' + # SolutionOutputFile: '$(Build.ArtifactStagingDirectory)/solution_unmanaged.zip' + # SolutionType: 'Unmanaged' + # + # - task: PowerPlatformImportSolution@2 + # displayName: 'Import into CI Dev' + # inputs: + # authenticationType: 'PowerPlatformSPN' + # PowerPlatformSPN: '' + # SolutionInputFile: '$(Build.ArtifactStagingDirectory)/solution_unmanaged.zip' + # AsyncOperation: true + # MaxAsyncWaitTime: 60 + # + # # Note: You still need to resolve the bot ID via Dataverse query or hardcode it. + # # The Build Tools tasks don't resolve bot IDs automatically. + + # Run the eval gate + - script: | + set -euo pipefail + set +x + + TOKEN_OUTPUT="$(Agent.TempDirectory)/updated-refresh-token" + + MCS_FLAG="" + if [ -n "${MCS_CONNECTION_ID:-}" ]; then + MCS_FLAG="--mcs-connection-id $MCS_CONNECTION_ID" + fi + + node scripts/eval-gate.mjs run \ + --config eval-config.json \ + --agent-id "$(RESOLVED_BOT_ID)" \ + $MCS_FLAG \ + --run-name "PR $(System.PullRequest.PullRequestNumber) @ $(Build.SourceVersion)" \ + --output "$(Build.ArtifactStagingDirectory)/eval-results.json" \ + --junit-output "$(Build.ArtifactStagingDirectory)/eval-results.junit.xml" \ + --token-output "$TOKEN_OUTPUT" + + echo "##vso[task.setvariable variable=TOKEN_OUTPUT_PATH]$TOKEN_OUTPUT" + displayName: 'Run Copilot Studio Eval' + env: + EVAL_REFRESH_TOKEN: $(copilot-studio-eval-refresh-token) + MCS_CONNECTION_ID: $(copilot-studio-eval-mcs-connection-id) + + # Publish test results to ADO Tests tab + - task: PublishTestResults@2 + displayName: 'Publish eval test results' + condition: always() + inputs: + testResultsFormat: 'JUnit' + testResultsFiles: '$(Build.ArtifactStagingDirectory)/eval-results.junit.xml' + testRunTitle: 'Copilot Studio Eval' + failTaskOnFailedTests: false + + # Write updated refresh token back to Key Vault + - task: AzureCLI@2 + displayName: 'Update refresh token in Key Vault' + condition: always() + inputs: + azureSubscription: '$(AZURE_SERVICE_CONNECTION)' + scriptType: 'bash' + scriptLocation: 'inlineScript' + inlineScript: | + set +x + TOKEN_FILE="$(Agent.TempDirectory)/updated-refresh-token" + if [ -f "$TOKEN_FILE" ]; then + az keyvault secret set \ + --vault-name "$(KEY_VAULT_NAME)" \ + --name "copilot-studio-eval-refresh-token" \ + --value "$(cat "$TOKEN_FILE")" \ + --output none + rm -f "$TOKEN_FILE" + echo "Refresh token updated in Key Vault and temp file cleaned up" + else + echo "No updated refresh token found — skipping" + fi + + # Publish eval results as artifact + - task: PublishBuildArtifacts@1 + displayName: 'Publish eval results' + condition: always() + inputs: + PathtoPublish: '$(Build.ArtifactStagingDirectory)' + ArtifactName: 'eval-$(Build.SourceVersion)' diff --git a/testing/evaluation/EvalGateADO/scripts/eval-gate.mjs b/testing/evaluation/EvalGateADO/scripts/eval-gate.mjs new file mode 100644 index 00000000..da7bf506 --- /dev/null +++ b/testing/evaluation/EvalGateADO/scripts/eval-gate.mjs @@ -0,0 +1,516 @@ +#!/usr/bin/env node + +/** + * eval-gate.mjs — Copilot Studio Evaluation API (PPAPI) client for CI/CD. + * + * Standalone script for running Copilot Studio evaluations as a pipeline gate. + * Uses a pre-cached MSAL refresh token (from Azure Key Vault) for unattended auth. + * + * Subcommands: + * node eval-gate.mjs run Start eval, poll, check results, exit 0/1 + * node eval-gate.mjs list-testsets List available test sets + * node eval-gate.mjs resolve-bot Resolve bot GUID from schema name in the target environment + * node eval-gate.mjs auth Interactive login (one-time setup) + * + * Options: + * --config Path to eval-config.json (default: ./eval-config.json) + * --environment-id Override environment ID from config + * --environment-url Override environment URL from config + * --agent-id Override agent ID from config (or auto-resolved from botSchemaName) + * --bot-schema-name Bot schema name (e.g., cr981_hotelfinder) — used to resolve agent ID dynamically + * --tenant-id Override tenant ID from config + * --client-id Override app registration client ID from config + * --testset-id Override test set ID from config + * --testset-name Match test set by display name + * --mcs-connection-id MCS connection ID for authenticated eval (connector actions, SharePoint, etc.) + * --threshold <0.0-1.0> Override pass threshold from config + * --run-name Display name for the eval run + * --output Write results JSON to this file + * --junit-output Write JUnit XML to this file (for ADO Tests tab) + * --token-output Write updated refresh token to this file + * + * Environment variables: + * EVAL_REFRESH_TOKEN Pre-cached MSAL refresh token (from Key Vault) + */ + +import { readFileSync, writeFileSync, existsSync } from "fs"; +import { PublicClientApplication } from "@azure/msal-node"; + +// --------------------------------------------------------------------------- +// CLI parsing +// --------------------------------------------------------------------------- + +function parseArgs() { + const args = process.argv.slice(2); + const parsed = { command: null, config: "./eval-config.json" }; + + for (let i = 0; i < args.length; i++) { + if (args[i].startsWith("--")) { + const key = args[i].slice(2).replace(/-([a-z])/g, (_, c) => c.toUpperCase()); + parsed[key] = args[++i]; + } else if (!parsed.command) { + parsed.command = args[i]; + } + } + + if (!parsed.command) { + die("Usage: eval-gate.mjs [options]"); + } + + return parsed; +} + +function loadConfig(args) { + let config = {}; + if (existsSync(args.config)) { + config = JSON.parse(readFileSync(args.config, "utf8")); + log(`Loaded config from ${args.config}`); + } + + // CLI flags override config file + return { + environmentId: args.environmentId || config.environmentId, + environmentUrl: args.environmentUrl || config.environmentUrl, + agentId: args.agentId || config.agentId, + botSchemaName: args.botSchemaName || config.botSchemaName, + tenantId: args.tenantId || config.tenantId, + clientId: args.clientId || config.clientId, + testSetId: args.testsetId || config.testSetId, + testSetName: args.testsetName || config.testSetName, + mcsConnectionId: args.mcsConnectionId || config.mcsConnectionId, + passThreshold: parseFloat(args.threshold || config.passThreshold || "0.8"), + runName: args.runName || `Eval ${new Date().toISOString()}`, + output: args.output, + junitOutput: args.junitOutput, + tokenOutput: args.tokenOutput, + }; +} + +function validate(config, ...required) { + for (const key of required) { + if (!config[key]) die(`Missing required config: ${key}. Set in eval-config.json or --${key.replace(/[A-Z]/g, c => `-${c.toLowerCase()}`)}`); + } +} + +// --------------------------------------------------------------------------- +// Logging +// --------------------------------------------------------------------------- + +function log(msg) { console.error(`[eval-gate] ${msg}`); } +function die(msg) { console.error(`[eval-gate] ERROR: ${msg}`); process.exit(1); } + +// --------------------------------------------------------------------------- +// Auth — MSAL refresh token flow +// --------------------------------------------------------------------------- + +const PP_API_SCOPE = "https://api.powerplatform.com/.default"; + +async function getToken(config) { + const refreshToken = process.env.EVAL_REFRESH_TOKEN; + if (!refreshToken) { + die("EVAL_REFRESH_TOKEN environment variable is not set. Run 'eval-gate.mjs auth' first, or inject from Key Vault."); + } + + const pca = new PublicClientApplication({ + auth: { + clientId: config.clientId, + authority: `https://login.microsoftonline.com/${config.tenantId}`, + }, + }); + + try { + const result = await pca.acquireTokenByRefreshToken({ + refreshToken, + scopes: [PP_API_SCOPE], + }); + + log(`Token acquired (expires ${result.expiresOn.toISOString()})`); + + // Write updated refresh token if available (restrictive permissions) + if (config.tokenOutput && result.refreshToken) { + writeFileSync(config.tokenOutput, result.refreshToken, { encoding: "utf8", mode: 0o600 }); + log(`Updated refresh token written to ${config.tokenOutput}`); + } + + return result.accessToken; + } catch (err) { + die(`Token acquisition failed: ${err.message}\nRefresh token may be expired. Re-run 'eval-gate.mjs auth' to get a new one.`); + } +} + +async function interactiveAuth(config) { + validate(config, "tenantId", "clientId"); + + const pca = new PublicClientApplication({ + auth: { + clientId: config.clientId, + authority: `https://login.microsoftonline.com/${config.tenantId}`, + }, + }); + + log("Starting device code flow..."); + await pca.acquireTokenByDeviceCode({ + scopes: [PP_API_SCOPE], + deviceCodeCallback: (response) => { + console.error("\n" + response.message + "\n"); + }, + }); + + // MSAL doesn't expose refresh tokens on the result object. + // Read from the in-memory cache instead. + const serialized = pca.getTokenCache().serialize(); + const parsed = JSON.parse(serialized); + const rtKeys = Object.keys(parsed.RefreshToken || {}); + + if (rtKeys.length === 0) { + die("No refresh token in cache. Ensure the app registration has 'Allow public client flows' set to Yes."); + } + + const refreshToken = parsed.RefreshToken[rtKeys[0]].secret; + console.error("\nAuthentication successful!\n"); + console.log(refreshToken); + console.error(`\nStore it in Key Vault with:`); + console.error(` az keyvault secret set --vault-name --name copilot-studio-eval-refresh-token --value ""\n`); + console.error(`Or pipe directly:`); + console.error(` node eval-gate.mjs auth --config eval-config.json | az keyvault secret set --vault-name --name copilot-studio-eval-refresh-token --value @-\n`); +} + +// --------------------------------------------------------------------------- +// PPAPI HTTP client +// --------------------------------------------------------------------------- + +const API_VERSION = "2024-10-01"; + +function baseUrl(config) { + return `https://api.powerplatform.com/copilotstudio/environments/${config.environmentId}/bots/${config.agentId}/api/makerevaluation`; +} + +async function ppApi(method, url, accessToken, body) { + const opts = { + method, + headers: { + Authorization: `Bearer ${accessToken}`, + "Content-Type": "application/json", + }, + signal: AbortSignal.timeout(30_000), + }; + if (body && method !== "GET") opts.body = JSON.stringify(body); + + const sep = url.includes("?") ? "&" : "?"; + const fullUrl = `${url}${sep}api-version=${API_VERSION}`; + + const res = await fetch(fullUrl, opts); + + if (!res.ok) { + const errBody = await res.text().catch(() => ""); + if (res.status === 401 || res.status === 403) die(`Auth failed (${res.status}). Re-run 'auth' to refresh token.\n${errBody.slice(0, 300)}`); + if (res.status === 409) die(`Conflict (409): An eval run is already in progress. Wait for it to complete.\n${errBody.slice(0, 300)}`); + if (res.status === 429) die(`Rate limited (429): Max 20 eval runs per bot per 24 hours.\n${errBody.slice(0, 300)}`); + die(`HTTP ${res.status}: ${errBody.slice(0, 500)}`); + } + + const text = await res.text(); + return text.trim() ? JSON.parse(text) : null; +} + +// --------------------------------------------------------------------------- +// Bot ID resolution — queries Dataverse for the bot GUID by schema name +// --------------------------------------------------------------------------- + +async function resolveBotId(config, accessToken) { + if (config.agentId && !config.agentId.startsWith("<")) return config.agentId; + + if (!config.botSchemaName || !config.environmentUrl) { + die("To auto-resolve bot ID, set both botSchemaName and environmentUrl in eval-config.json.\nAlternatively, set agentId directly via --agent-id."); + } + + validate(config, "environmentId"); + log(`Resolving bot ID for schema name: ${config.botSchemaName}`); + + // Query Dataverse OData API — requires a Dataverse-scoped token + const dvScope = `${config.environmentUrl}.default`; + const pca = new PublicClientApplication({ + auth: { + clientId: config.clientId, + authority: `https://login.microsoftonline.com/${config.tenantId}`, + }, + }); + + let dvToken; + try { + const result = await pca.acquireTokenByRefreshToken({ + refreshToken: process.env.EVAL_REFRESH_TOKEN, + scopes: [dvScope], + }); + dvToken = result.accessToken; + } catch (err) { + die(`Cannot get Dataverse token for bot resolution: ${err.message}\nEnsure your app registration has Dynamics CRM > user_impersonation permission.\nOr set agentId directly via --agent-id.`); + } + + const schemaName = config.botSchemaName.replace(/'/g, "''"); + const odataFilter = encodeURIComponent(`schemaname eq '${schemaName}'`); + const odataUrl = `${config.environmentUrl}api/data/v9.2/bots?$filter=${odataFilter}&$select=botid,name,schemaname`; + const res = await fetch(odataUrl, { + headers: { + Authorization: `Bearer ${dvToken}`, + Accept: "application/json", + "OData-MaxVersion": "4.0", + "OData-Version": "4.0", + }, + signal: AbortSignal.timeout(30_000), + }); + + if (!res.ok) { + const errBody = await res.text().catch(() => ""); + die(`Dataverse query failed (${res.status}): ${errBody.slice(0, 300)}\nSet agentId directly via --agent-id.`); + } + + const data = await res.json(); + const bots = data?.value || []; + + if (bots.length === 0) { + die(`Bot '${config.botSchemaName}' not found in environment. Ensure the solution has been imported.`); + } + + const bot = bots[0]; + log(`Resolved bot ID: ${bot.botid} (${bot.name})`); + config.agentId = bot.botid; + return bot.botid; +} + +async function resolveTestSetId(config, accessToken) { + if (config.testSetId && !config.testSetId.startsWith("<")) return config.testSetId; + + log("Resolving test set ID..."); + const data = await ppApi("GET", `${baseUrl(config)}/testsets`, accessToken); + const testsets = (data?.value || []).filter(ts => ts.state === "Ready" || ts.state === "Active"); + + if (testsets.length === 0) { + die("No test sets found for this bot. Create one in Copilot Studio (Evaluate tab)."); + } + + if (config.testSetName) { + const match = testsets.find(ts => ts.displayName === config.testSetName); + if (!match) { + const available = testsets.map(ts => ` ${ts.displayName} (${ts.id})`).join("\n"); + die(`Test set '${config.testSetName}' not found.\nAvailable:\n${available}`); + } + log(`Resolved test set: ${match.displayName} (${match.id}, ${match.totalTestCases} cases)`); + config.testSetId = match.id; + return match.id; + } + + const ts = testsets[0]; + log(`Using test set: ${ts.displayName} (${ts.id}, ${ts.totalTestCases} cases)`); + config.testSetId = ts.id; + return ts.id; +} + +// --------------------------------------------------------------------------- +// Subcommands +// --------------------------------------------------------------------------- + +async function cmdResolvBot(config) { + validate(config, "environmentId", "tenantId", "clientId"); + const token = await getToken(config); + const botId = await resolveBotId(config, token); + console.log(JSON.stringify({ botId, botSchemaName: config.botSchemaName }, null, 2)); +} + +async function cmdListTestsets(config) { + validate(config, "environmentId", "tenantId", "clientId"); + const token = await getToken(config); + if (!config.agentId || config.agentId.startsWith("<")) await resolveBotId(config, token); + const data = await ppApi("GET", `${baseUrl(config)}/testsets`, token); + const testsets = data?.value || []; + + console.log(JSON.stringify({ testsets: testsets.map(ts => ({ + id: ts.id, + displayName: ts.displayName, + totalTestCases: ts.totalTestCases, + state: ts.state, + }))}, null, 2)); +} + +async function cmdRun(config) { + validate(config, "environmentId", "tenantId", "clientId"); + const token = await getToken(config); + if (!config.agentId || config.agentId.startsWith("<")) await resolveBotId(config, token); + if (!config.testSetId || config.testSetId.startsWith("<")) await resolveTestSetId(config, token); + + // Step 1: Start eval run + log(`Starting eval run: "${config.runName}"`); + log(`Test set: ${config.testSetId} | Threshold: ${(config.passThreshold * 100).toFixed(0)}%`); + + const runBody = { + runOnPublishedBot: false, + evaluationRunName: config.runName, + }; + if (config.mcsConnectionId) { + runBody.mcsConnectionId = config.mcsConnectionId; + log(`Using MCS connection: ${config.mcsConnectionId}`); + } else { + log("No mcsConnectionId — authenticated actions/knowledge sources won't work during eval"); + } + + const startData = await ppApi("POST", `${baseUrl(config)}/testsets/${config.testSetId}/run`, token, runBody); + + const runId = startData.runId; + log(`Run started: ${runId}`); + + // Step 2: Poll until complete (max 10 min, every 20s) + let state = "unknown"; + for (let i = 0; i < 30; i++) { + await sleep(20_000); + const status = await ppApi("GET", `${baseUrl(config)}/testruns/${runId}`, token); + state = status.state; + const processed = status.testCasesProcessed || 0; + const total = status.totalTestCases || 0; + log(`Progress: ${processed}/${total} (${state})`); + + if (["Completed", "Failed", "Cancelled", "Abandoned"].includes(state)) break; + } + + if (state !== "Completed") { + die(`Eval run did not complete (state: ${state})`); + } + + // Step 3: Fetch results + const results = await ppApi("GET", `${baseUrl(config)}/testruns/${runId}`, token); + + if (config.output) { + writeFileSync(config.output, JSON.stringify(results, null, 2), "utf8"); + log(`Results written to ${config.output}`); + } + + // Step 4: Evaluate pass/fail + const testCases = results.testCasesResults || []; + const total = testCases.length; + + function getOverallStatus(tc) { + const metrics = tc.metricsResults || []; + if (metrics.length === 0) return "Error"; + const allPass = metrics.every(m => m.result?.status === "Pass"); + const anyError = metrics.some(m => m.result?.status === "Error"); + if (allPass) return "Pass"; + if (anyError) return "Error"; + return "Fail"; + } + + const passed = testCases.filter(tc => getOverallStatus(tc) === "Pass").length; + const failed = total - passed; + const passRate = total > 0 ? passed / total : 0; + + console.log(""); + console.log("========================================="); + console.log(` EVAL RESULTS: ${passed}/${total} passed (${(passRate * 100).toFixed(1)}%)`); + console.log(` Threshold: ${(config.passThreshold * 100).toFixed(0)}%`); + console.log(` Verdict: ${passRate >= config.passThreshold ? "PASS" : "FAIL"}`); + console.log("========================================="); + + if (failed > 0) { + console.log("\nFailed test cases:"); + for (const tc of testCases.filter(t => getOverallStatus(t) !== "Pass")) { + const failedMetrics = (tc.metricsResults || []) + .filter(m => m.result?.status !== "Pass") + .map(m => `${m.type}: ${m.result?.status} (${Object.entries(m.result?.data || {}).filter(([,v]) => v === "No").map(([k]) => k).join(", ") || m.result?.errorReason || "unknown"})`) + .join("; "); + console.log(` - ${tc.testCaseId}: ${getOverallStatus(tc)} [${failedMetrics}]`); + } + } + + // Calculate run duration + const startTime = results.startTime ? new Date(results.startTime) : null; + const endTime = results.endTime ? new Date(results.endTime) : null; + const durationSec = (startTime && endTime) ? ((endTime - startTime) / 1000).toFixed(1) : "0"; + + // Write JUnit XML for ADO Tests tab + if (config.junitOutput) { + const escXml = (s) => String(s || "").replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); + + function metricSummary(tc) { + return (tc.metricsResults || []).map(m => { + const data = m.result?.data || {}; + const dims = Object.entries(data).map(([k, v]) => `${k}: ${v}`).join(", "); + return `[${m.type}] status=${m.result?.status} | ${dims}${m.result?.errorReason ? ` | error: ${m.result.errorReason}` : ""}${m.result?.aiResultReason ? `\nAI reason: ${m.result.aiResultReason}` : ""}`; + }).join("\n"); + } + + const perTestDuration = total > 0 ? (parseFloat(durationSec) / total).toFixed(1) : "0"; + + const tcXml = testCases.map(tc => { + const name = escXml(tc.testCaseId); + const status = getOverallStatus(tc); + const details = escXml(metricSummary(tc)); + + if (status === "Pass") { + return ` \n ${details}\n `; + } + + const failedMetrics = (tc.metricsResults || []) + .filter(m => m.result?.status !== "Pass") + .map(m => { + const failedDims = Object.entries(m.result?.data || {}).filter(([,v]) => v === "No").map(([k]) => k).join(", "); + return `${m.type}: ${m.result?.status} (${failedDims || m.result?.errorReason || "unknown"})`; + }).join("; "); + + return ` \n ${details}\n `; + }).join("\n"); + + const xml = ` + + +${tcXml} + +`; + + writeFileSync(config.junitOutput, xml, "utf8"); + log(`JUnit XML written to ${config.junitOutput}`); + } + + // Output summary as JSON + console.log("\n" + JSON.stringify({ + runId, + runName: config.runName, + total, + passed, + failed, + passRate, + threshold: config.passThreshold, + verdict: passRate >= config.passThreshold ? "PASS" : "FAIL", + }, null, 2)); + + if (passRate < config.passThreshold) { + process.exit(1); + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function sleep(ms) { return new Promise(r => setTimeout(r, ms)); } + +// --------------------------------------------------------------------------- +// Main +// --------------------------------------------------------------------------- + +const args = parseArgs(); +const config = loadConfig(args); + +switch (args.command) { + case "run": + await cmdRun(config); + break; + case "list-testsets": + await cmdListTestsets(config); + break; + case "resolve-bot": + await cmdResolvBot(config); + break; + case "auth": + await interactiveAuth(config); + break; + default: + die(`Unknown command: ${args.command}\nCommands: run, list-testsets, resolve-bot, auth`); +} diff --git a/testing/evaluation/EvalGateADO/scripts/package.json b/testing/evaluation/EvalGateADO/scripts/package.json new file mode 100644 index 00000000..5e4ddc5f --- /dev/null +++ b/testing/evaluation/EvalGateADO/scripts/package.json @@ -0,0 +1,9 @@ +{ + "name": "eval-gate", + "version": "1.0.0", + "type": "module", + "description": "Copilot Studio Evaluation API client for CI/CD pipelines", + "dependencies": { + "@azure/msal-node": "^2.0.0" + } +} diff --git a/testing/evaluation/README.md b/testing/evaluation/README.md new file mode 100644 index 00000000..22fd08e9 --- /dev/null +++ b/testing/evaluation/README.md @@ -0,0 +1,16 @@ +--- +title: Evaluation +parent: Testing +nav_order: 2 +has_children: true +has_toc: false +--- +# Evaluation + +Automated evaluation samples for Copilot Studio agents using the [Evaluation REST API](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-rest-api). + +## Contents + +| Sample | Description | +|--------|-------------| +| [EvalGateADO/](./EvalGateADO/) | Azure DevOps pipeline that uses the Copilot Studio Evaluation API as a PR quality gate |