aws / aws/amazon-sagemaker-examples

ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code

Open
#4,456 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am using the Sagemaker [Pytorch Estimator][1] based on a custom docker image stored in AWS ECR.

```python
from sagemaker.pytorch.estimator import PyTorch

role = "arn:..."

estimator = PyTorch(
image_uri="1...ecr...amazonaws.com/...:prototype",
git_config={"repo": "https://github.com/celsofranssa/LightningPrototype.git", "branch": "sagemaker"},
entry_point="main.py",
role=role,
region="us-...",
instance_type="local", # ml.g4dn.2xlarge
instance_count=1,
volume_size=225,
hyperparameters=hparams
)
estimator.fit()
```
Sagemaker correctly clones the sources from GitHub and performs the checkout into the specified branch.

**The Bug:**
However, it only copies the `main.py` to `/opt/ml/code` inside the container instead of the holy-cloned source code, which causing `ModuleNotFoundError: No module named 'source'`:
```python
Traceback (most recent call last):
2y9byzwyxr-algo-1-reuoy | File "/opt/ml/code/main.py", line 15, in
2y9byzwyxr-algo-1-reuoy | from source.helper.EvalHelper import EvalHelper
2y9byzwyxr-algo-1-reuoy | ModuleNotFoundError: No module named 'source'
```

Logging the `/opt/ml/code` content only shows the `main.py`:
```
print(f"Content: {os.listdir(os.getcwd())}")
['main.py']
```

[1]: https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html

Contributor guide

Open the contributing guide

Research direction

Start with the PyTorch Estimator configuration in the issue and compare its git_config and entry_point behavior with the linked SageMaker documentation. Reproduce the run and inspect /opt/ml/code and the ModuleNotFoundError for source. Done means establishing whether the cloned source should be copied there and documenting or fixing the responsible behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.