aws / aws/amazon-sagemaker-examples
Advanced example distributed tensorflow training exits with mpirun exit code 1 during evaluations
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Advanced example for distributed Tensorflow Mask R-CNN model training exits with mpirun exit code 1 during evaluation part of training. This is caused by an incompatibility with the version of Python numpy package being installed by Tensorpack install.
The fix is to change container/Dockerfile to fix the version of numpy to 1.17.5 and NOT use --ignore-installed flag with Tensorpack install. I will commit a fix on a branch and issue a pull request.
Contributor guide
Research direction
Start with container/Dockerfile and inspect the Tensorpack installation and NumPy dependency versions. Reproduce the advanced distributed TensorFlow Mask R-CNN example through evaluation, then verify that the container builds with NumPy 1.17.5 and that evaluation no longer exits with mpirun exit code 1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dockerfile, numpy, python, tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100