Skip to content

Add two new models to the nightly benchmark run - #2139

Merged
Fan Yang (fanyang-mono) merged 3 commits into
mainfrom
dev/yangfan/add_new_models
May 2, 2026
Merged

Add two new models to the nightly benchmark run#2139
Fan Yang (fanyang-mono) merged 3 commits into
mainfrom
dev/yangfan/add_new_models

Conversation

@fanyang-mono

Copy link
Copy Markdown
Contributor

Description

Add Opus 4.7 and GPT 5.5 to the nightly benchmark run.

Co-authored-by: Copilot <copilot@github.com>
Copilot AI review requested due to automatic review settings May 1, 2026 23:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates the Azure DevOps nightly benchmark runner to include additional LLM models in the MSBench run set.

Changes:

  • Adds claude-opus-4.7 and gpt-5.5 to the default model list for nightly benchmark runs.
  • Extends Key Vault secret retrieval to include CAPI integration credentials and exports them as environment variables for the benchmark runs.
Show a summary per file
File Description
pipelines/scripts/Invoke-RunBenchmarks.ps1 Updates the model list for nightly runs and adds retrieval/usage of additional secrets needed by the benchmark execution.

Copilot's findings

Comments suppressed due to low confidence (1)

pipelines/scripts/Invoke-RunBenchmarks.ps1:94

  • The catch block message still says "Failed to retrieve GitHub PAT from KeyVault", but this try block now retrieves multiple secrets (GitHub PAT + CAPI integration id + CAPI HMAC). Update the error text (and ideally the preceding Write-Host) to reflect all required secrets so failures are actionable.
    try {
        Write-Host "Retrieving GitHub PAT from KeyVault $vaultName secret $secretNameGhPAT"
        $pat = az keyvault secret show --vault-name $vaultName --name $secretNameGhPAT --query value -o tsv
        $capiId = az keyvault secret show --vault-name $vaultName --name $secretNameCAPIID --query value -o tsv
        $capiHmac = az keyvault secret show --vault-name $vaultName --name $secretNameCAPIHMAC --query value -o tsv

        if (!$pat) {
            throw "Secret $secretNameGhPAT not found in KeyVault $vaultName."
        }

        $env:GITHUB_MCP_SERVER_TOKEN = $pat
        $env:CAPI_INTEGRATION_ID = $capiId
        $env:CAPI_HMAC_KEY = $capiHmac
        # Log the secrets as secret variables to avoid exposing them in logs
        if ($pipelineRun) {
            Write-Host "##vso[task.setsecret]$pat"
            Write-Host "##vso[task.setsecret]$capiId"
            Write-Host "##vso[task.setsecret]$capiHmac"
        }
    }
    catch {
        throw "Failed to retrieve GitHub PAT from KeyVault: $_"
    }
  • Files reviewed: 1/1 changed files
  • Comments generated: 3

Comment thread pipelines/scripts/Invoke-RunBenchmarks.ps1
Comment thread pipelines/scripts/Invoke-RunBenchmarks.ps1 Outdated
Comment thread pipelines/scripts/Invoke-RunBenchmarks.ps1 Outdated
Fan Yang (fanyang-mono) and others added 2 commits May 1, 2026 19:56
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@fanyang-mono
Fan Yang (fanyang-mono) merged commit 771a666 into main May 2, 2026
9 checks passed
@fanyang-mono
Fan Yang (fanyang-mono) deleted the dev/yangfan/add_new_models branch May 2, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants