aws / aws/sagemaker-pytorch-inference-toolkit
ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code
- Dominant language
- Python
- Stars
- 143
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
I am using the Sagemaker [Pytorch Estimator][1] based on a custom docker image stored in AWS ECR.
```python
from sagemaker.pytorch.estimator import PyTorch
role = "arn:..."
estimator = PyTorch(
image_uri="1...ecr...amazonaws.com/...:prototype",
git_config={"repo": "https://github.com/celsofranssa/LightningPrototype.git", "branch": "sagemaker"},
entry_point="main.py",
role=role,
region="us-...",
instance_type="local", # ml.g4dn.2xlarge
instance_count=1,
volume_size=225,
hyperparameters=hparams
)
estimator.fit()
```
Sagemaker correctly clones the sources from GitHub and performs the checkout into the specified branch.
**The Bug:**
However, it only copies the `main.py` to `/opt/ml/code` inside the container instead of the holy-cloned source code, which causing `ModuleNotFoundError: No module named 'source'`:
```python
Traceback (most recent call last):
2y9byzwyxr-algo-1-reuoy | File "/opt/ml/code/main.py", line 15, in
2y9byzwyxr-algo-1-reuoy | from source.helper.EvalHelper import EvalHelper
2y9byzwyxr-algo-1-reuoy | ModuleNotFoundError: No module named 'source'
```
Logging the `/opt/ml/code` content only shows the `main.py`:
```
print(f"Content: {os.listdir(os.getcwd())}")
['main.py']
```
[1]: https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html
Contributor guide
Research direction
Reproduce the PyTorch Estimator setup from the issue with git_config, entry_point="main.py", and estimator.fit(), then inspect the contents of /opt/ml/code in the custom container. Compare the checked-out repository with the files staged there; done means the cloned source is available in /opt/ml/code and the import from source.helper.EvalHelper succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, git, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100