Azure / Azure/azureml-examples

Mlflow error on pytorch.log_model but model is saved

Open
#937 0 comments 0 reactions 0 assignees View on GitHub
example issue
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

## Which example? Describe the issue

example: [CIFAR pytorch distributed](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/pytorch/cifar-distributed)

description: model training shows completed, model is saved as well but driver logs (`70_driver_log..`.) for the model saving driver has:

`ERROR mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /tmp/tmpvthoxt0n/model/data, flavor: pytorch)
Traceback (most recent call last):
File "/azureml-envs/pytorch-1.9/lib/python3.7/site-packages/mlflow/utils/environment.py", line 194, in infer_pip_requirements
return _infer_requirements(model_uri, flavor)
File "/azureml-envs/pytorch-1.9/lib/python3.7/site-packages/mlflow/utils/requirements_utils.py", line 306, in _infer_requirements
_MODULES_TO_PACKAGES = importlib_metadata.packages_distributions()
AttributeError: module 'importlib_metadata' has no attribute 'packages_distributions'`

## Additional context

Tried with variations to the environment in `job.yml: azureml:AzureML-pytorch-1.9-ubuntu18.04-py37-cuda11-gpu:11 and azureml:AzureML-pytorch-1.9-ubuntu18.04-py37-cuda11-gpu:6`. Same outcome.

Contributor guide

Open the contributing guide

Research direction

Start with the CIFAR PyTorch distributed example and its job.yml, then inspect the 70_driver_log.. output around model saving and the reported MLflow traceback. Compare the two listed AzureML PyTorch environments and determine whether the model-save operation is complete despite the error; done means the example's behavior and expected driver-log outcome are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.