aws / aws/sagemaker-training-toolkit

ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code

Open
#200 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

I am using the Sagemaker [Pytorch Estimator][1] based on a custom docker image stored in AWS ECR.

```python
from sagemaker.pytorch.estimator import PyTorch

role = "arn:..."

estimator = PyTorch(
image_uri="1...ecr...amazonaws.com/...:prototype",
git_config={"repo": "https://github.com/celsofranssa/LightningPrototype.git", "branch": "sagemaker"},
entry_point="main.py",
role=role,
region="us-...",
instance_type="local", # ml.g4dn.2xlarge
instance_count=1,
volume_size=225,
hyperparameters=hparams
)
estimator.fit()
```
Sagemaker correctly clones the sources from GitHub and performs the checkout into the specified branch.

**The Bug:**
However, it only copies the `main.py` to `/opt/ml/code` inside the container instead of the holy-cloned source code, which causing `ModuleNotFoundError: No module named 'source'`:
```python
Traceback (most recent call last):
2y9byzwyxr-algo-1-reuoy | File "/opt/ml/code/main.py", line 15, in
2y9byzwyxr-algo-1-reuoy | from source.helper.EvalHelper import EvalHelper
2y9byzwyxr-algo-1-reuoy | ModuleNotFoundError: No module named 'source'
```

Logging the `/opt/ml/code` content only shows the `main.py`:
```
print(f"Content: {os.listdir(os.getcwd())}")
['main.py']
```

[1]: https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html

Contributor guide

Open the contributing guide

Research direction

Start with the PyTorch Estimator configuration and its git_config handling, then reproduce the run using the custom ECR image and the provided repository and branch. Inspect how the cloned checkout is staged into /opt/ml/code; done means the source directory is available there so main.py can import source.helper without ModuleNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.