Azure / Azure/azure-cli-extensions

Unclear error in AzureML Computer job

Open
#3,574 9 comments 0 reactions 0 assignees View on GitHub
ADO bug extension/ml Machine Learning ML-MLOps Service Attention
Dominant language
Python
Stars
454
Forks
1.7k
Avg merge
2d 19h
Merged PRs (30d)
64

Description

- If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at [Azure/azure-cli](https://github.com/Azure/azure-cli/issues)

### Extension name (the extension in question)
I am using the Azure ML 2.0 CLI.

### Description of issue (in as much detail as possible)

I built a container that I upload to ACR with my code and artifacts in place. I can run the container locally and get its to start successfully. The entry point starts from `/opt/main.py`. However, when I run it into Azure ML, it abruptly breaks with the status below.

> AzureMLCompute job failed.
> JobFailed: Submitted script failed with a non-zero exit code; see the driver log file for details.
> Reason: Job failed with non-zero exit Code

My YAML that I am launching the CLI with is as follows:

> $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
> command: >-
> python /opt/main.py --set
> training-data={inputs.training}
> validation-data={inputs.validation}
> testing-data={inputs.testing}
> inputs:
> training:
> data:
> path: https://****.blob.core.windows.net/modeldata/data1.csv
> mode: mount
> validation:
> data:
> path: https://****.blob.core.windows.net/modeldata/data2.csv
> mode: mount
> testing:
> data:
> path: https://****.blob.core.windows.net/modeldata/data3.csv
> mode: mount
> environment:
> name: sklearn-train
> version: 1
> docker:
> image: ****.azurecr.io/****/train-image:latest
>
>
> compute:
> target: azureml:/subscriptions/c24881db-d5d4-482e-b582-d17f74d863ef/resourceGroups/****/providers/Microsoft.MachineLearningServices/workspaces/****/computes/****-test
> experiment_name: sk_learn
> description: Train a sklearn model

When I go to the driver logs, this is all that's there.

2021/07/01 23:00:04 Starting App Insight Logger for task: runTaskLet
2021/07/01 23:00:04 Version: 3.0.01632.0003 Branch: .SourceBranch Commit: 4b96fb0
2021/07/01 23:00:04 Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/info
2021/07/01 23:00:04 Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/status
[2021-07-01T23:00:04.789779] Entering context manager injector.

-----

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.