Azure / Azure/azureml-examples
Mlflow error on pytorch.log_model but model is saved
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
## Which example? Describe the issue
example: [CIFAR pytorch distributed](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/pytorch/cifar-distributed)
description: model training shows completed, model is saved as well but driver logs (`70_driver_log..`.) for the model saving driver has:
`ERROR mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /tmp/tmpvthoxt0n/model/data, flavor: pytorch)
Traceback (most recent call last):
File "/azureml-envs/pytorch-1.9/lib/python3.7/site-packages/mlflow/utils/environment.py", line 194, in infer_pip_requirements
return _infer_requirements(model_uri, flavor)
File "/azureml-envs/pytorch-1.9/lib/python3.7/site-packages/mlflow/utils/requirements_utils.py", line 306, in _infer_requirements
_MODULES_TO_PACKAGES = importlib_metadata.packages_distributions()
AttributeError: module 'importlib_metadata' has no attribute 'packages_distributions'`
## Additional context
Tried with variations to the environment in `job.yml: azureml:AzureML-pytorch-1.9-ubuntu18.04-py37-cuda11-gpu:11 and azureml:AzureML-pytorch-1.9-ubuntu18.04-py37-cuda11-gpu:6`. Same outcome.
Contributor guide
Research direction
Start with the CIFAR PyTorch distributed example and its job.yml, then inspect the 70_driver_log.. output around model saving and the reported MLflow traceback. Compare the two listed AzureML PyTorch environments and determine whether the model-save operation is complete despite the error; done means the example's behavior and expected driver-log outcome are clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100