Azure / Azure/MachineLearningNotebooks

AzureML Compute Job failed to start

Open
#1,389 4 comments 0 reactions 0 assignees View on GitHub
ADO Compute product-issue
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Unable to use dockerfile to create an Environment in the `explain-model-on-amlcompute.ipynb` notebook.

The following is the code

```
from azureml.core import Workspace, Environment
myenv = Environment(name="rapidsenv")

myenv.register(workspace=ws)
myenv.docker.enabled=True

dockerfile = r"""
FROM rapidsai/rapidsai-cloud-ml:0.17-cuda11.0-base-ubuntu18.04-py3.8
RUN echo "success" # I'd like to install different packages here
"""
myenv.docker.base_image = None
myenv.docker.base_dockerfile = dockerfile
myenv.python.user_managed_dependencies=True

from azureml.core import Run
from azureml.core import ScriptRunConfig

src = ScriptRunConfig(source_directory=project_folder,
script='train_explain.py',
compute_target=gpu_cluster,
environment=myenv)
run = experiment.submit(config=src)
run
```

This is what shows up on the Portal.

```
AzureMLCompute job failed.
JobContainerConfigFailed: Container configuration failed unexpectedly
JobContainerConfigFailed: Container configuration failed unexpectedly
err: Docker exec failure when attempting to execute 'containerSetup' task. Error: exit status 126. Indicates a command was found but not executable, likely due to permission issues or missing dependencies.
Reason: Docker exec failure when attempting to execute 'containerSetup' task. Error: exit status 126. Indicates a command was found but not executable, likely due to permission issues or missing dependencies.
Info: Failed to prepare an environment for the job execution: Job environment preparation failed on 10.0.0.4 with err exit status 1.
```

How do I resolve this issue?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with explain-model-on-amlcompute.ipynb and run the provided Environment and ScriptRunConfig example against the GPU compute target. Use the Portal's JobContainerConfigFailed and containerSetup exit-status-126 messages to trace environment preparation; done means the notebook job starts successfully with the Dockerfile-based environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, docker, jupyter-notebook, python
Domain
cloud, devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.