Azure / Azure/MachineLearningNotebooks

Unable to pull custom docker image in Compute Cluster from Azure private registry

Open
#1,733 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have created a new machine learning environment with a connected ACR (without admin keys enabled).
We also have created a User assigned managed identity to enable managed identity authentication for the compute clusters. This identity has the ACRPull role assigned on the connected ACR.

However, when we try the following Python code as seen in the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-managed-identities?tabs=python#pull-docker-base-image-to-machine-learning-compute-cluster-for-training-as-is), it does not work:
```
env = Environment(name="private-acr")
env.docker.base_image = "as01weuacrom4vosf3mpux7.azurecr.io/custom:v1"
env.python.user_managed_dependencies = True
```
```
We get the following error message:
AzureMLCompute job failed.
FailedPullingImage: Unable to pull docker image
imageName: as01weuacrom4vosf3mpux7.azurecr.io/custom:v1
error: Run docker command to pull public image failed with error: Error response from daemon: Head "https://as01weuacrom4vosf3mpux7.azurecr.io/v2/custom/manifests/v1": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
.
Reason: Error response from daemon: Head "https://as01weuacrom4vosf3mpux7.azurecr.io/v2/custom/manifests/v1": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

Info: Failed to setup runtime for job execution: Job environment preparation failed on 10.235.22.6 with err exit status 1.
```
We also tried to build the image in ACR, but it also does not authenticate automatically.
The only way we can get this to work is when we enable ACR access keys, but this obviously is not a preferred solution.

Please not that all resources are behind a vnet, including the compute cluster which has public ip disabled.

Can you please advice what to do next, or what we can try?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.