`az containerapp job start --image` replaces the requested image with the quickstart image when `--registry-identity system` is used
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
When starting an existing Azure Container Apps job with both:
```powershell
--image
--registry-identity system
```
the execution does not use the image supplied through `--image`.
Instead, Azure CLI submits the following public quickstart image in the execution template:
```powershell
mcr.microsoft.com/k8se/quickstart:latest
```
The existing job is already configured with:
* A system-assigned managed identity
* A private Azure Container Registry
* The registry identity set to `system`
* The requested private image in the stored job template
Therefore, no identity bootstrap should be required when starting an execution.
### Related command
[az containerapp job start](https://learn.microsoft.com/en-us/cli/azure/containerapp/job?view=azure-cli-latest#az-containerapp-job-start)
### Errors
No CLI exception or non-zero exit code is returned.
`az containerapp job start` completes successfully with exit code `0` and creates an execution. However, inspecting the resulting execution shows that the image supplied through `--image` was replaced.
The execution was started using:
```powershell
$execution = az containerapp job start `
--resource-group "" `
--name "" `
--image ".azurecr.io/:" `
--container-name "" `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv
```
The resulting execution was inspected using:
```powershell
az containerapp job execution show `
--resource-group "" `
--name "" `
--job-execution-name $execution `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc
```
The inspection command returned:
```json
[
{
"cpu": 0.25,
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"memory": "0.5Gi",
"name": ""
}
]
```
The command therefore reports success, but the execution uses `mcr.microsoft.com/k8se/quickstart:latest` instead of the private image explicitly supplied through `--image`.
### Issue script & Debug output
```powershell
$ErrorActionPreference = "Stop"
$SubscriptionId = ""
$ResourceGroup = ""
$EnvironmentName = ""
$AcrName = ""
$WorkloadProfileName = "Consumption"
$Suffix = Get-Date -Format "MMddHHmmss"
$JobName = "cli-job-start-repro-$Suffix"
$ContainerName = "repro"
$Repository = "cli-job-start-repro"
$Tag = $Suffix
$SourceImage = "mcr.microsoft.com/k8se/quickstart-jobs:latest"
$ExecutionName = $null
$JobCreated = $false
$ImageImported = $false
function Assert-AzSuccess {
param([string]$Operation)
if ($LASTEXITCODE -ne 0) {
throw "$Operation failed with exit code $LASTEXITCODE."
}
}
try {
Write-Host "Selecting the test subscription"
az account set `
--subscription $SubscriptionId
Assert-AzSuccess "Selecting the subscription"
$LoginServerOutput = az acr show `
--name $AcrName `
--query loginServer `
--output tsv
Assert-AzSuccess "Reading the ACR login server"
$LoginServer = ($LoginServerOutput | Out-String).Trim()
$PrivateImage = "${LoginServer}/${Repository}:${Tag}"
Write-Host "Importing a small test image into the private ACR"
az acr import `
--name $AcrName `
--source $SourceImage `
--image "${Repository}:${Tag}" `
--force `
--output none
Assert-AzSuccess "Importing the test image"
$ImageImported = $true
Write-Host "Creating a temporary job on the Consumption workload profile"
az containerapp job create `
--resource-group $ResourceGroup `
--name $JobName `
--environment $EnvironmentName `
--trigger-type Manual `
--replica-timeout 300 `
--replica-retry-limit 0 `
--replica-completion-count 1 `
--parallelism 1 `
--image $PrivateImage `
--container-name $ContainerName `
--cpu 0.25 `
--memory 0.5Gi `
--workload-profile-name $WorkloadProfileName `
--mi-system-assigned `
--registry-server $LoginServer `
--registry-identity system `
--output none
Assert-AzSuccess "Creating the temporary job"
$JobCreated = $true
Write-Host ""
Write-Host "Stored job configuration"
Write-Host ""
az containerapp job show `
--resource-group $ResourceGroup `
--name $JobName `
--query '{identityType:identity.type,registries:properties.configuration.registries,workloadProfile:properties.workloadProfileName,containers:properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}}' `
--output jsonc `
--debug
Assert-AzSuccess "Reading the stored job configuration"
Write-Host ""
Write-Host "Starting the execution with --image and --registry-identity system"
Write-Host ""
$ExecutionOutput = az containerapp job start `
--resource-group $ResourceGroup `
--name $JobName `
--image $PrivateImage `
--container-name $ContainerName `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv `
--debug
Assert-AzSuccess "Starting the job execution"
$ExecutionName = ($ExecutionOutput | Out-String).Trim()
Write-Host ""
Write-Host "Execution name: $ExecutionName"
Write-Host ""
Write-Host "Resulting execution template"
Write-Host ""
az containerapp job execution show `
--resource-group $ResourceGroup `
--name $JobName `
--job-execution-name $ExecutionName `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc `
--debug
Assert-AzSuccess "Reading the execution template"
}
finally {
if ($ExecutionName) {
Write-Host ""
Write-Host "Stopping the temporary execution"
az containerapp job stop `
--resource-group $ResourceGroup `
--name $JobName `
--job-execution-name $ExecutionName `
--only-show-errors `
--output none
}
if ($JobCreated) {
Write-Host "Deleting the temporary job"
az containerapp job delete `
--resource-group $ResourceGroup `
--name $JobName `
--yes `
--only-show-errors `
--output none
}
if ($ImageImported) {
Write-Host "Deleting the temporary ACR image"
az acr repository delete `
--name $AcrName `
--image "${Repository}:${Tag}" `
--yes `
--only-show-errors `
--output none
}
}
```
The complete reproduction script above was run with Azure CLI `2.87.0` and the `containerapp` extension `1.3.0b4`.
The existing job was successfully created with the requested private image:
```json
{
"containers": [
{
"cpu": 0.25,
"image": ".azurecr.io/cli-job-start-repro:",
"memory": "0.5Gi",
"name": "repro"
}
],
"identityType": "SystemAssigned",
"registries": [
{
"identity": "system",
"server": ".azurecr.io"
}
],
"workloadProfile": "Consumption"
}
```
The following command was then run:
```powershell
$ExecutionOutput = az containerapp job start `
--resource-group "" `
--name "" `
--image ".azurecr.io/cli-job-start-repro:" `
--container-name "repro" `
--cpu 0.25 `
--memory 0.5Gi `
--registry-identity system `
--query name `
--output tsv `
--debug
```
Relevant sanitized debug output:
```json
CommandName: containerapp job start
ParameterSetName: --resource-group --name --image --container-name --cpu --memory --registry-identity --query --output --debug
Request URL:
https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.App/jobs//start?api-version=
Request method:
POST
Authorization:
Request body:
{
"containers": [
{
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"name": "repro",
"resources": {
"cpu": 0.25,
"memory": "0.5Gi"
}
}
]
}
```
The request body shows that Azure CLI replaced the image supplied through `--image` before sending the request to the Container Apps service.
The resulting execution was inspected using:
```powershell
az containerapp job execution show `
--resource-group "" `
--name "" `
--job-execution-name "" `
--query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
--output jsonc `
--debug
```
The execution contained:
```json
[
{
"cpu": 0.25,
"image": "mcr.microsoft.com/k8se/quickstart:latest",
"memory": "0.5Gi",
"name": "repro"
}
]
```
The command exited successfully with exit code `0`, but the wrong image was submitted and executed.
### Expected behavior
The execution should use the image explicitly supplied through `--image`:
```text
.azurecr.io/:
```
The job's existing registry configuration and system-assigned identity should be used to authenticate the image pull.
At minimum, `--registry-identity system` must not silently replace the requested image.
### Environment Summary
```json
PS> az version
{
"azure-cli": "2.87.0",
"azure-cli-core": "2.87.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"containerapp": "1.3.0b4"
}
}
PS> az extension show `
--name containerapp `
--query '{name:name, version:version, path:path}' `
--output json
{
"name": "containerapp",
"path": "C:\\Users\\\\.azure\\cliextensions\\containerapp",
"version": "1.3.0b4"
}
```
### Additional context
#### Suspected cause
The current implementation in https://github.com/Azure/azure-cli/blob/fc0ff8f8cf62d9f6ea12e28125e8e32458701e3d/src/azure-cli/azure/cli/command_modules/containerapp/custom.py#L1575-L1577
contains this logic in the job-start path:
```python
# If no image is provided, fetch the existing job's image
if image is not None:
container_def["image"] = image if not is_registry_msi_system(registry_identity) else HELLO_WORLD_IMAGE
```
This appears to reuse the public-placeholder-image logic needed when initially creating a resource with a new system-assigned identity.
That bootstrap can make sense during job creation:
1. Create the job with a public image.
2. Obtain the newly created system identity's principal ID.
3. Grant the identity permission to pull from ACR.
4. Update the job to use the requested private image.
However, `job start` operates on an existing job whose identity and registry permissions already exist. It also does not perform a subsequent update that restores the requested image. The quickstart image therefore becomes the final execution image.
#### Suggested fix
The immediate fix appears to be to preserve the requested image unconditionally:
```python
if image is not None:
container_def["image"] = image
```
Additionally, please consider removing or deprecating `--registry-identity` from `az containerapp job start`.
Registry authentication is persistent job configuration and can already be configured through job creation or the job registry commands. If execution-level registry identity overrides are not supported by the service API, the CLI should reject this option with a clear error rather than silently changing the image.
For backward compatibility, a possible migration would be:
1. Stop replacing the image with `HELLO_WORLD_IMAGE`.
2. Emit a warning or validation error when `--registry-identity` is passed to `job start`.
3. Deprecate and later remove the argument from this command.
Contributor guide
Assessment
This issue has not been assessed yet.