Azure / Azure/azureml-examples

Cannot invoke a batch deployment

Open
#2,614 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

MacOS

### Version Information

azure-cli 2.50.0 *

core 2.50.0 *
telemetry 1.0.8 *

Extensions:
ml 2.19.1

Dependencies:
msal 1.22.0
azure-mgmt-resource 23.1.0b2

Python (Darwin) 3.10.13 (main, Aug 24 2023, 22:48:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]

### Steps to reproduce

When trying to invoke your azureml batch deployment example mnist for the torch model it fails.
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-batch-model-deployments?view=azureml-api-2&tabs=cli#about-this-example

`JOB_NAME=$(az ml batch-endpoint invoke --name mnist-batch-dv --input https://azuremlexampledata.blob.core.windows.net/data/mnist/sample --input-type uri_folder --query name -o tsv)`

### Expected behavior

The job was supposed to run with the example data according to your tutorial.

### Actual behavior

Getting the error:

`Execution failed. User process '/azureml-envs/azureml_7594b3b934a904695f71542edf30f209/bin/python' exited with status code 42. Please check log file 'user_logs/std_log_0.txt' for error details. Error: Traceback (most recent call last): File "driver/amlbi_main.py", line 275, in main() File "driver/amlbi_main.py", line 226, in main sys.exit(exitcode_candidate) SystemExit: 42`

`Azure Machine Learning Batch Inference Start
[2023-09-02 09:59:18.451846] No started flag set. Skip creating started flag.
Azure Machine Learning Batch Inference End
Cleaning up all outstanding Run operations, waiting 300.0 seconds
2 items cleaning up...
Cleanup took 0.14278292655944824 seconds
Traceback (most recent call last):
File "driver/amlbi_main.py", line 275, in
main()
File "driver/amlbi_main.py", line 226, in main
sys.exit(exitcode_candidate)
SystemExit: 42`

### Addition information

It worked ok until last week (22nd August per my last check).

I noticed that something is wrong with the input to `driver/amlbi_main.py` which I can't find since its not open code.

It is getting the following `--model $AZUREML_DATAREFERENCE_score_model/../ ` argument .

But two weeks ago the argument was:
`--model_name *** --model_version 1`

Contributor guide

Open the contributing guide

Research direction

Start with the batch model deployments tutorial and the reported `az ml batch-endpoint invoke` command for the MNIST Torch example. Inspect the `user_logs/std_log_0.txt` output and compare the reported `--model` argument with the earlier `--model_name` and `--model_version` arguments. Done means the documented invocation runs the example data without exit status 42.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cli, cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.