Azure / Azure/azureml-examples
Cannot invoke a batch deployment
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
MacOS
### Version Information
azure-cli 2.50.0 *
core 2.50.0 *
telemetry 1.0.8 *
Extensions:
ml 2.19.1
Dependencies:
msal 1.22.0
azure-mgmt-resource 23.1.0b2
Python (Darwin) 3.10.13 (main, Aug 24 2023, 22:48:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]
### Steps to reproduce
When trying to invoke your azureml batch deployment example mnist for the torch model it fails.
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-batch-model-deployments?view=azureml-api-2&tabs=cli#about-this-example
`JOB_NAME=$(az ml batch-endpoint invoke --name mnist-batch-dv --input https://azuremlexampledata.blob.core.windows.net/data/mnist/sample --input-type uri_folder --query name -o tsv)`
### Expected behavior
The job was supposed to run with the example data according to your tutorial.
### Actual behavior
Getting the error:
`Execution failed. User process '/azureml-envs/azureml_7594b3b934a904695f71542edf30f209/bin/python' exited with status code 42. Please check log file 'user_logs/std_log_0.txt' for error details. Error: Traceback (most recent call last): File "driver/amlbi_main.py", line 275, in main() File "driver/amlbi_main.py", line 226, in main sys.exit(exitcode_candidate) SystemExit: 42`
`Azure Machine Learning Batch Inference Start
[2023-09-02 09:59:18.451846] No started flag set. Skip creating started flag.
Azure Machine Learning Batch Inference End
Cleaning up all outstanding Run operations, waiting 300.0 seconds
2 items cleaning up...
Cleanup took 0.14278292655944824 seconds
Traceback (most recent call last):
File "driver/amlbi_main.py", line 275, in
main()
File "driver/amlbi_main.py", line 226, in main
sys.exit(exitcode_candidate)
SystemExit: 42`
### Addition information
It worked ok until last week (22nd August per my last check).
I noticed that something is wrong with the input to `driver/amlbi_main.py` which I can't find since its not open code.
It is getting the following `--model $AZUREML_DATAREFERENCE_score_model/../ ` argument .
But two weeks ago the argument was:
`--model_name *** --model_version 1`
Contributor guide
Research direction
Start with the batch model deployments tutorial and the reported `az ml batch-endpoint invoke` command for the MNIST Torch example. Inspect the `user_logs/std_log_0.txt` output and compare the reported `--model` argument with the earlier `--model_name` and `--model_version` arguments. Done means the documented invocation runs the example data without exit status 42.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cli, cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100