Azure / Azure/azure-cli

`az containerapp job start --image` replaces the requested image with the quickstart image when `--registry-identity system` is used

Open
#33,548 2 comments 0 reactions 0 assignees View on GitHub
act-observability-squad Auto-Assign bug ContainerApp customer-reported Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

When starting an existing Azure Container Apps job with both:

```powershell
--image
--registry-identity system
```

the execution does not use the image supplied through `--image`.

Instead, Azure CLI submits the following public quickstart image in the execution template:

```powershell
mcr.microsoft.com/k8se/quickstart:latest
```

The existing job is already configured with:

* A system-assigned managed identity
* A private Azure Container Registry
* The registry identity set to `system`
* The requested private image in the stored job template

Therefore, no identity bootstrap should be required when starting an execution.

### Related command

[az containerapp job start](https://learn.microsoft.com/en-us/cli/azure/containerapp/job?view=azure-cli-latest#az-containerapp-job-start)

### Errors

No CLI exception or non-zero exit code is returned.

`az containerapp job start` completes successfully with exit code `0` and creates an execution. However, inspecting the resulting execution shows that the image supplied through `--image` was replaced.

The execution was started using:

```powershell
$execution = az containerapp job start `
--resource-group "" `
--name "" `
--image ".azurecr.io/:" `
--container-name "" `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv
```

The resulting execution was inspected using:

```powershell
az containerapp job execution show `
--resource-group "" `
--name "" `
--job-execution-name $execution `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc
```

The inspection command returned:

```json
[
{
"cpu": 0.25,
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"memory": "0.5Gi",
"name": ""
}
]
```

The command therefore reports success, but the execution uses `mcr.microsoft.com/k8se/quickstart:latest` instead of the private image explicitly supplied through `--image`.

### Issue script & Debug output

```powershell
$ErrorActionPreference = "Stop"

$SubscriptionId = ""
$ResourceGroup = ""
$EnvironmentName = ""
$AcrName = ""
$WorkloadProfileName = "Consumption"

$Suffix = Get-Date -Format "MMddHHmmss"
$JobName = "cli-job-start-repro-$Suffix"
$ContainerName = "repro"
$Repository = "cli-job-start-repro"
$Tag = $Suffix
$SourceImage = "mcr.microsoft.com/k8se/quickstart-jobs:latest"

$ExecutionName = $null
$JobCreated = $false
$ImageImported = $false

function Assert-AzSuccess {
param([string]$Operation)

if ($LASTEXITCODE -ne 0) {
throw "$Operation failed with exit code $LASTEXITCODE."
}
}

try {
Write-Host "Selecting the test subscription"

az account set `
--subscription $SubscriptionId

Assert-AzSuccess "Selecting the subscription"

$LoginServerOutput = az acr show `
--name $AcrName `
--query loginServer `
--output tsv

Assert-AzSuccess "Reading the ACR login server"

$LoginServer = ($LoginServerOutput | Out-String).Trim()
$PrivateImage = "${LoginServer}/${Repository}:${Tag}"

Write-Host "Importing a small test image into the private ACR"

az acr import `
--name $AcrName `
--source $SourceImage `
--image "${Repository}:${Tag}" `
--force `
--output none

Assert-AzSuccess "Importing the test image"
$ImageImported = $true

Write-Host "Creating a temporary job on the Consumption workload profile"

az containerapp job create `
--resource-group $ResourceGroup `
--name $JobName `
--environment $EnvironmentName `
--trigger-type Manual `
--replica-timeout 300 `
--replica-retry-limit 0 `
--replica-completion-count 1 `
--parallelism 1 `
--image $PrivateImage `
--container-name $ContainerName `
--cpu 0.25 `
--memory 0.5Gi `
--workload-profile-name $WorkloadProfileName `
--mi-system-assigned `
--registry-server $LoginServer `
--registry-identity system `
--output none

Assert-AzSuccess "Creating the temporary job"
$JobCreated = $true

Write-Host ""
Write-Host "Stored job configuration"
Write-Host ""

az containerapp job show `
--resource-group $ResourceGroup `
--name $JobName `
--query '{identityType:identity.type,registries:properties.configuration.registries,workloadProfile:properties.workloadProfileName,containers:properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}}' `
--output jsonc `
--debug

Assert-AzSuccess "Reading the stored job configuration"

Write-Host ""
Write-Host "Starting the execution with --image and --registry-identity system"
Write-Host ""

$ExecutionOutput = az containerapp job start `
--resource-group $ResourceGroup `
--name $JobName `
--image $PrivateImage `
--container-name $ContainerName `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv `
--debug

Assert-AzSuccess "Starting the job execution"

$ExecutionName = ($ExecutionOutput | Out-String).Trim()

Write-Host ""
Write-Host "Execution name: $ExecutionName"
Write-Host ""
Write-Host "Resulting execution template"
Write-Host ""

az containerapp job execution show `
--resource-group $ResourceGroup `
--name $JobName `
--job-execution-name $ExecutionName `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc `
--debug

Assert-AzSuccess "Reading the execution template"
}
finally {
if ($ExecutionName) {
Write-Host ""
Write-Host "Stopping the temporary execution"

az containerapp job stop `
--resource-group $ResourceGroup `
--name $JobName `
--job-execution-name $ExecutionName `
--only-show-errors `
--output none
}

if ($JobCreated) {
Write-Host "Deleting the temporary job"

az containerapp job delete `
--resource-group $ResourceGroup `
--name $JobName `
--yes `
--only-show-errors `
--output none
}

if ($ImageImported) {
Write-Host "Deleting the temporary ACR image"

az acr repository delete `
--name $AcrName `
--image "${Repository}:${Tag}" `
--yes `
--only-show-errors `
--output none
}
}
```

The complete reproduction script above was run with Azure CLI `2.87.0` and the `containerapp` extension `1.3.0b4`.

The existing job was successfully created with the requested private image:

```json
{
"containers": [
{
"cpu": 0.25,
"image": ".azurecr.io/cli-job-start-repro:",
"memory": "0.5Gi",
"name": "repro"
}
],
"identityType": "SystemAssigned",
"registries": [
{
"identity": "system",
"server": ".azurecr.io"
}
],
"workloadProfile": "Consumption"
}
```

The following command was then run:

```powershell
$ExecutionOutput = az containerapp job start `
--resource-group "" `
--name "" `
--image ".azurecr.io/cli-job-start-repro:" `
--container-name "repro" `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv `
--debug
```

Relevant sanitized debug output:

```json
CommandName: containerapp job start
ParameterSetName: --resource-group --name --image --container-name --cpu --memory --registry-identity --query --output --debug

Request URL:
https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.App/jobs//start?api-version=

Request method:
POST

Authorization:

Request body:
{
"containers": [
{
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"name": "repro",
"resources": {
"cpu": 0.25,
"memory": "0.5Gi"
}
}
]
}
```

The request body shows that Azure CLI replaced the image supplied through `--image` before sending the request to the Container Apps service.

The resulting execution was inspected using:

```powershell
az containerapp job execution show `
--resource-group "" `
--name "" `
--job-execution-name "" `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc `
--debug
```

The execution contained:

```json
[
{
"cpu": 0.25,
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"memory": "0.5Gi",
"name": "repro"
}
]
```

The command exited successfully with exit code `0`, but the wrong image was submitted and executed.

### Expected behavior

The execution should use the image explicitly supplied through `--image`:

```text
.azurecr.io/:
```

The job's existing registry configuration and system-assigned identity should be used to authenticate the image pull.

At minimum, `--registry-identity system` must not silently replace the requested image.

### Environment Summary

```json
PS> az version
{
"azure-cli": "2.87.0",
"azure-cli-core": "2.87.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"containerapp": "1.3.0b4"
}
}
PS> az extension show `
--name containerapp `
--query '{name:name, version:version, path:path}' `
--output json
{
"name": "containerapp",
"path": "C:\\Users\\\\.azure\\cliextensions\\containerapp",
"version": "1.3.0b4"
}
```

### Additional context

#### Suspected cause

The current implementation in https://github.com/Azure/azure-cli/blob/fc0ff8f8cf62d9f6ea12e28125e8e32458701e3d/src/azure-cli/azure/cli/command_modules/containerapp/custom.py#L1575-L1577

contains this logic in the job-start path:

```python
# If no image is provided, fetch the existing job's image
if image is not None:
container_def["image"] = image if not is_registry_msi_system(registry_identity) else HELLO_WORLD_IMAGE

```

This appears to reuse the public-placeholder-image logic needed when initially creating a resource with a new system-assigned identity.

That bootstrap can make sense during job creation:

1. Create the job with a public image.
2. Obtain the newly created system identity's principal ID.
3. Grant the identity permission to pull from ACR.
4. Update the job to use the requested private image.

However, `job start` operates on an existing job whose identity and registry permissions already exist. It also does not perform a subsequent update that restores the requested image. The quickstart image therefore becomes the final execution image.

#### Suggested fix

The immediate fix appears to be to preserve the requested image unconditionally:

```python
if image is not None:
container_def["image"] = image
```

Additionally, please consider removing or deprecating `--registry-identity` from `az containerapp job start`.

Registry authentication is persistent job configuration and can already be configured through job creation or the job registry commands. If execution-level registry identity overrides are not supported by the service API, the CLI should reject this option with a clear error rather than silently changing the image.

For backward compatibility, a possible migration would be:

1. Stop replacing the image with `HELLO_WORLD_IMAGE`.
2. Emit a warning or validation error when `--registry-identity` is passed to `job start`.
3. Deprecate and later remove the argument from this command.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.