Skip to content

feat: replace azure-validate AZCLI/Bicep validation steps with a shared script #2856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Tom Meschter (tmeschter) merged 2 commits into from
Jul 17, 2026
Merged

feat: replace azure-validate AZCLI/Bicep validation steps with a shared script #2856

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
68aba54
feat: replace azure-validate AZCLI/Bicep validation steps with a shar…
tmeschter Jul 15, 2026
d4155c6
fix: address PR review feedback on validate-deployment scripts
tmeschter Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions evals/azure-validate/e2e-eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,53 @@ stimuli:
- name: "(?i)^(bash|powershell|pwsh)$"
command: "(?i)azd provision --no-prompt"

# Jest: "runs the validate-deployment script for a standalone Bicep (az CLI) project"
Comment thread
tmeschter marked this conversation as resolved.
# No azure.yaml → azure-prepare selects the Bicep/AZCLI recipe, whose validation is the
# shared validate-deployment.{sh,ps1} helper. The agent must RUN that script (not azd,
# not raw az deployment create).
- name: "Runs validate-deployment Script - Standalone Bicep (az CLI)"
prompt: "My static status page is ready. I deploy it directly with the Azure CLI (az deployment) using my Bicep templates in ./infra — I do NOT use azd. Please prepare and validate it for deployment to my current subscription in eastus2."
environment:
files:
- src: fixture/bicep-cli/index.html
dest: index.html
- src: fixture/bicep-cli/infra/main.bicep
dest: infra/main.bicep
- src: fixture/bicep-cli/infra/resources.bicep
dest: infra/resources.bicep
- src: fixture/bicep-cli/infra/main.parameters.json
dest: infra/main.parameters.json
constraints:
max_turns: 50
tags:
type: integration
tier: full
cost: llm
area: behavior
skill: azure-validate
followUp:
- "Continue with recommended options until complete."
earlyTerminate: '[{"type":"skill-call","skill":"azure-deploy"},{"type":"tool-call-match","toolPattern":"bash|powershell|pwsh|run_in_terminal","argsPattern":"validate-deployment\\.(sh|ps1)"},{"type":"tool-call-match","toolPattern":"bash|powershell|pwsh|run_in_terminal","argsPattern":"az\\s+deployment\\b.*\\b(create|up)\\b"}]'
graders:
# azure-validate WAS invoked; azure-deploy was NOT
- type: skill-invocation
config:
required:
- azure-validate
disallowed:
- azure-deploy
# The shared validate-deployment script WAS run; no actual deployment happened
- type: tool-calls
config:
required:
- name: "(?i)^(bash|powershell|pwsh)$"
command: "(?i)validate-deployment\\.(sh|ps1)"
disallowed:
- name: "(?i)^(bash|powershell|pwsh)$"
command: "(?i)azd\\s+(up|deploy)\\b"
- name: "(?i)^(bash|powershell|pwsh)$"
command: "(?i)az\\s+deployment\\b.*\\bcreate\\b"

# ═══════════════════════════════════════════
# brownfield-dotnet-validate
# Source: describe("brownfield-dotnet-validate")
Expand Down
15 changes: 15 additions & 0 deletions evals/azure-validate/fixture/bicep-cli/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<!DOCTYPE html>
<html lang="en">
<head><meta charset="UTF-8"><title>Status Page</title><style>
body { font-family: sans-serif; max-width: 800px; margin: 0 auto; padding: 2rem; }
h1 { color: #333; }
.status { border: 1px solid #ddd; padding: 1rem; margin: 1rem 0; border-radius: 4px; }
</style></head>
<body>
<h1>Service Status</h1>
<section>
<div class="status"><h3>API</h3><p>Operational</p></div>
<div class="status"><h3>Web</h3><p>Operational</p></div>
</section>
</body>
</html>
32 changes: 32 additions & 0 deletions evals/azure-validate/fixture/bicep-cli/infra/main.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
targetScope = 'subscription'

@minLength(1)
@maxLength(64)
@description('Name of the environment that can be used as part of naming resource convention')
param environmentName string

@minLength(1)
@description('Primary location for all resources')
param location string

var tags = { 'azd-env-name': environmentName }
var resourceToken = toLower(uniqueString(subscription().id, environmentName, location))

resource rg 'Microsoft.Resources/resourceGroups@2022-09-01' = {
name: 'rg-${environmentName}'
location: location
tags: tags
}

module web './resources.bicep' = {
name: 'web'
scope: rg
params: {
location: location
tags: tags
resourceToken: resourceToken
}
}

output AZURE_LOCATION string = location
output AZURE_TENANT_ID string = tenant().tenantId
12 changes: 12 additions & 0 deletions evals/azure-validate/fixture/bicep-cli/infra/main.parameters.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"environmentName": {
"value": "statuspage"
},
"location": {
"value": "eastus2"
}
}
}
18 changes: 18 additions & 0 deletions evals/azure-validate/fixture/bicep-cli/infra/resources.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
@description('Location for all resources')
param location string = resourceGroup().location

param tags object = {}
param resourceToken string

resource staticWebApp 'Microsoft.Web/staticSites@2022-09-01' = {
name: 'swa-${resourceToken}'
location: location
tags: union(tags, { 'azd-service-name': 'web' })
sku: {
name: 'Free'
tier: 'Free'
}
properties: {}
}

output STATIC_WEB_APP_URL string = staticWebApp.properties.defaultHostname
92 changes: 31 additions & 61 deletions plugin/skills/azure-validate/references/recipes/azcli/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,84 +9,54 @@ Validation steps for Azure CLI deployments.

## Validation Steps

- [ ] 1. Azure CLI Installation
- [ ] 2. Authentication
- [ ] 3. Bicep Compilation
- [ ] 4. Template Validation
- [ ] 5. What-If Preview
- [ ] 6. Docker Build (if containerized)
- [ ] 7. Azure Policy Validation
- [ ] 1. Core Validation (CLI, auth, build, validate, what-if) — run [`validate-deployment` script](../scripts/validate-deployment.sh)
- [ ] 2. Docker Build (if containerized)
- [ ] 3. Azure Policy Validation

## Validation Details

### 1. Azure CLI Installation
### 1. Core Validation Script

Verify Azure CLI is installed:
The core validation checks are a fixed, deterministic sequence. Run the shared
**validate-deployment** helper instead of executing and parsing each command by hand. It
confirms the Azure CLI is installed and authenticated, compiles the Bicep template
(`az bicep build`), validates it against the target scope (`az deployment ... validate`),
and runs a what-if preview — printing a compact PASS/FAIL summary plus a what-if change
count (Create/Modify/Delete).

```bash
az version
```

**If not installed:**
```
mcp_azure_mcp_extension_cli_install(cli-type: "az")
```

### 2. Authentication
- Bash: [`../scripts/validate-deployment.sh`](../scripts/validate-deployment.sh)
- PowerShell: [`../scripts/validate-deployment.ps1`](../scripts/validate-deployment.ps1)

```bash
az account show
```
**Subscription scope:**

**If not logged in:**
```bash
az login
../scripts/validate-deployment.sh --scope sub --location <location>
```

**Set subscription:**
```bash
az account set --subscription <subscription-id>
```powershell
../scripts/validate-deployment.ps1 -Scope sub -Location <location>
```

### 3. Bicep Compilation
**Resource group scope:**

```bash
az bicep build --file ./infra/main.bicep
../scripts/validate-deployment.sh --scope group --resource-group <rg-name>
```

### 4. Template Validation

```bash
# Subscription scope
az deployment sub validate \
--location <location> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json

# Resource group scope
az deployment group validate \
--resource-group <rg-name> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json
```powershell
../scripts/validate-deployment.ps1 -Scope group -ResourceGroup <rg-name>
```

### 5. What-If Preview
Defaults: `--template ./infra/main.bicep`, `--parameters ./infra/main.parameters.json`
(skipped if absent). Pass `--subscription <id>` to target a specific subscription.

```bash
# Subscription scope
az deployment sub what-if \
--location <location> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json

# Resource group scope
az deployment group what-if \
--resource-group <rg-name> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json
```
**Interpreting results:**

- `OVERALL: PASS` — all five checks passed; record the summary in Section 7 (Validation Proof).
- Any step `FAIL` — the script prints the failing command's error. Remediate:
- **Authenticated** fails → `az login`, then `az account set --subscription <id>`.
- **Azure CLI installed** fails → install via `mcp_azure_mcp_extension_cli_install(cli-type: "az")`.
- Otherwise see [Error handling](./errors.md).

### 6. Docker Build (if containerized)
### 2. Docker Build (if containerized)

**Before building**, validate the Docker build context:

Expand All @@ -105,7 +75,7 @@ npm install --package-lock-only
docker build -t <image>:test ./src/<service>
```

### 7. Azure Policy Validation
### 3. Azure Policy Validation

See [Policy Validation Guide](../../policy-validation.md) for instructions on retrieving and validating Azure policies for your subscription.

Expand Down
77 changes: 31 additions & 46 deletions plugin/skills/azure-validate/references/recipes/bicep/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,83 +10,68 @@ Validation steps for standalone Bicep deployments.

## Validation Steps

- [ ] 1. Bicep Compilation
- [ ] 2. Template Validation
- [ ] 3. What-If Preview
- [ ] 4. Authentication
- [ ] 5. Linting (optional)
- [ ] 6. Azure Policy Validation
- [ ] 1. Core Validation (CLI, auth, build, validate, what-if) — run [`validate-deployment` script](../scripts/validate-deployment.sh)
- [ ] 2. Linting (optional)
- [ ] 3. Azure Policy Validation

## Validation Details

### 1. Bicep Compilation
### 1. Core Validation Script

```bash
az bicep build --file ./infra/main.bicep
```
The core validation checks are a fixed, deterministic sequence — identical to the AZCLI
recipe. Run the shared **validate-deployment** helper instead of executing and parsing each
command by hand. It confirms the Azure CLI is installed and authenticated, compiles the Bicep
template (`az bicep build`), validates it against the target scope (`az deployment ...
validate`), and runs a what-if preview — printing a compact PASS/FAIL summary plus a what-if
change count (Create/Modify/Delete).

**Pass:** No output (compiles cleanly)
**Fail:** Shows line numbers and errors
- Bash: [`../scripts/validate-deployment.sh`](../scripts/validate-deployment.sh)
- PowerShell: [`../scripts/validate-deployment.ps1`](../scripts/validate-deployment.ps1)

### 2. Template Validation
**Subscription scope:**

```bash
# Subscription scope
az deployment sub validate \
--location <location> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json

# Resource group scope
az deployment group validate \
--resource-group <rg-name> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json
../scripts/validate-deployment.sh --scope sub --location <location>
```
```powershell
../scripts/validate-deployment.ps1 -Scope sub -Location <location>
```

### 3. What-If Preview
**Resource group scope:**

```bash
az deployment sub what-if \
--location <location> \
--template-file ./infra/main.bicep \
--parameters ./infra/main.parameters.json
```

**Expected output:**
../scripts/validate-deployment.sh --scope group --resource-group <rg-name>
```
Resource and property changes are indicated with these symbols:
+ Create
~ Modify
- Delete
```powershell
../scripts/validate-deployment.ps1 -Scope group -ResourceGroup <rg-name>
```

### 4. Authentication
Defaults: `--template ./infra/main.bicep`, `--parameters ./infra/main.parameters.json`
(skipped if absent). Pass `--subscription <id>` to target a specific subscription.

```bash
az account show
```
**Interpreting results:**

- `OVERALL: PASS` — all checks passed; record the summary in Section 7 (Validation Proof).
- Any step `FAIL` — the script prints the failing command's error. If **Authenticated** fails,
run `az login`. Otherwise see [Error handling](./errors.md).

### 5. Linting (optional)
### 2. Linting (optional)

Use Bicep linter rules:

```bash
az bicep lint --file ./infra/main.bicep
```

### 6. Azure Policy Validation
### 3. Azure Policy Validation

See [Policy Validation Guide](../../policy-validation.md) for instructions on retrieving and validating Azure policies for your subscription.

## Checklist

| Check | Command | Pass |
|-------|---------|------|
| Bicep compiles | `az bicep build` | ☐ |
| Template valid | `az deployment validate` | ☐ |
| What-if passes | `az deployment what-if` | ☐ |
| Auth valid | `az account show` | ☐ |
| Core validation (CLI, auth, build, validate, what-if) | `validate-deployment` script | ☐ |
| Policies validated | MCP Policy tool | ☐ |

## References
Expand Down
Loading
Loading