aws / aws/amazon-sagemaker-examples

Advanced example distributed tensorflow training exits with mpirun exit code 1 during evaluations

Open
#1,014 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Advanced example for distributed Tensorflow Mask R-CNN model training exits with mpirun exit code 1 during evaluation part of training. This is caused by an incompatibility with the version of Python numpy package being installed by Tensorpack install.

The fix is to change container/Dockerfile to fix the version of numpy to 1.17.5 and NOT use --ignore-installed flag with Tensorpack install. I will commit a fix on a branch and issue a pull request.

Contributor guide

Open the contributing guide

Research direction

Start with container/Dockerfile and inspect the Tensorpack installation and NumPy dependency versions. Reproduce the advanced distributed TensorFlow Mask R-CNN example through evaluation, then verify that the container builds with NumPy 1.17.5 and that evaluation no longer exits with mpirun exit code 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
dockerfile, numpy, python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.