Add two new models to the nightly benchmark run - #2139
Merged
Conversation
Co-authored-by: Copilot <copilot@github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request updates the Azure DevOps nightly benchmark runner to include additional LLM models in the MSBench run set.
Changes:
- Adds
claude-opus-4.7andgpt-5.5to the default model list for nightly benchmark runs. - Extends Key Vault secret retrieval to include CAPI integration credentials and exports them as environment variables for the benchmark runs.
Show a summary per file
| File | Description |
|---|---|
pipelines/scripts/Invoke-RunBenchmarks.ps1 |
Updates the model list for nightly runs and adds retrieval/usage of additional secrets needed by the benchmark execution. |
Copilot's findings
Comments suppressed due to low confidence (1)
pipelines/scripts/Invoke-RunBenchmarks.ps1:94
- The catch block message still says "Failed to retrieve GitHub PAT from KeyVault", but this try block now retrieves multiple secrets (GitHub PAT + CAPI integration id + CAPI HMAC). Update the error text (and ideally the preceding Write-Host) to reflect all required secrets so failures are actionable.
try {
Write-Host "Retrieving GitHub PAT from KeyVault $vaultName secret $secretNameGhPAT"
$pat = az keyvault secret show --vault-name $vaultName --name $secretNameGhPAT --query value -o tsv
$capiId = az keyvault secret show --vault-name $vaultName --name $secretNameCAPIID --query value -o tsv
$capiHmac = az keyvault secret show --vault-name $vaultName --name $secretNameCAPIHMAC --query value -o tsv
if (!$pat) {
throw "Secret $secretNameGhPAT not found in KeyVault $vaultName."
}
$env:GITHUB_MCP_SERVER_TOKEN = $pat
$env:CAPI_INTEGRATION_ID = $capiId
$env:CAPI_HMAC_KEY = $capiHmac
# Log the secrets as secret variables to avoid exposing them in logs
if ($pipelineRun) {
Write-Host "##vso[task.setsecret]$pat"
Write-Host "##vso[task.setsecret]$capiId"
Write-Host "##vso[task.setsecret]$capiHmac"
}
}
catch {
throw "Failed to retrieve GitHub PAT from KeyVault: $_"
}
- Files reviewed: 1/1 changed files
- Comments generated: 3
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
JasonYeMSFT (JasonYeMSFT)
approved these changes
May 2, 2026
This was referenced May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add Opus 4.7 and GPT 5.5 to the nightly benchmark run.