aws / aws/sagemaker-training-toolkit
ModuleNotFoundError: Sagemaker only copies entry_point file to /opt/ml/code/ instead of the holy-cloned source code
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
I am using the Sagemaker [Pytorch Estimator][1] based on a custom docker image stored in AWS ECR.
```python
from sagemaker.pytorch.estimator import PyTorch
role = "arn:..."
estimator = PyTorch(
image_uri="1...ecr...amazonaws.com/...:prototype",
git_config={"repo": "https://github.com/celsofranssa/LightningPrototype.git", "branch": "sagemaker"},
entry_point="main.py",
role=role,
region="us-...",
instance_type="local", # ml.g4dn.2xlarge
instance_count=1,
volume_size=225,
hyperparameters=hparams
)
estimator.fit()
```
Sagemaker correctly clones the sources from GitHub and performs the checkout into the specified branch.
**The Bug:**
However, it only copies the `main.py` to `/opt/ml/code` inside the container instead of the holy-cloned source code, which causing `ModuleNotFoundError: No module named 'source'`:
```python
Traceback (most recent call last):
2y9byzwyxr-algo-1-reuoy | File "/opt/ml/code/main.py", line 15, in
2y9byzwyxr-algo-1-reuoy | from source.helper.EvalHelper import EvalHelper
2y9byzwyxr-algo-1-reuoy | ModuleNotFoundError: No module named 'source'
```
Logging the `/opt/ml/code` content only shows the `main.py`:
```
print(f"Content: {os.listdir(os.getcwd())}")
['main.py']
```
[1]: https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html
Contributor guide
Research direction
Start with the PyTorch Estimator configuration and its git_config handling, then reproduce the run using the custom ECR image and the provided repository and branch. Inspect how the cloned checkout is staged into /opt/ml/code; done means the source directory is available there so main.py can import source.helper without ModuleNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100