aws / aws/amazon-sagemaker-examples
ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I am using the Sagemaker [Pytorch Estimator][1] based on a custom docker image stored in AWS ECR.
```python
from sagemaker.pytorch.estimator import PyTorch
role = "arn:..."
estimator = PyTorch(
image_uri="1...ecr...amazonaws.com/...:prototype",
git_config={"repo": "https://github.com/celsofranssa/LightningPrototype.git", "branch": "sagemaker"},
entry_point="main.py",
role=role,
region="us-...",
instance_type="local", # ml.g4dn.2xlarge
instance_count=1,
volume_size=225,
hyperparameters=hparams
)
estimator.fit()
```
Sagemaker correctly clones the sources from GitHub and performs the checkout into the specified branch.
**The Bug:**
However, it only copies the `main.py` to `/opt/ml/code` inside the container instead of the holy-cloned source code, which causing `ModuleNotFoundError: No module named 'source'`:
```python
Traceback (most recent call last):
2y9byzwyxr-algo-1-reuoy | File "/opt/ml/code/main.py", line 15, in
2y9byzwyxr-algo-1-reuoy | from source.helper.EvalHelper import EvalHelper
2y9byzwyxr-algo-1-reuoy | ModuleNotFoundError: No module named 'source'
```
Logging the `/opt/ml/code` content only shows the `main.py`:
```
print(f"Content: {os.listdir(os.getcwd())}")
['main.py']
```
[1]: https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html
Contributor guide
Research direction
Start with the PyTorch Estimator configuration in the issue and compare its git_config and entry_point behavior with the linked SageMaker documentation. Reproduce the run and inspect /opt/ml/code and the ModuleNotFoundError for source. Done means establishing whether the cloned source should be copied there and documenting or fixing the responsible behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100