aws / aws/sagemaker-training-toolkit
Silent Failure if custom image puts something into /opt/ml/code
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
Hi, I was making a new Docker image for training:
```
FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04
COPY src/requirements.txt /opt/ml/code/requirements.txt
RUN pip install --no-cache-dir -r /opt/ml/code/requirements.txt
```
And I found that when I do that, my training image could no longer find the files that usually get copied in when the container runs. I traced it back to this line, which checks if the /opt/ml/code folder exists, and if it exists at all it just skips the step that copies over the sourcedir.tar.gz file from that URI.
Should the logic be changed so that it doesn't skip downloading the file, or maybe at least it should give a warning that it's skipping the download?
https://github.com/aws/sagemaker-training-toolkit/blob/628166c157751ae2a46fddc11a7a8cac765fb22c/src/sagemaker_training/files.py#L134
Contributor guide
Research direction
Start at src/sagemaker_training/files.py around line 134 and trace the condition that handles an existing /opt/ml/code directory and the sourcedir.tar.gz URI. Reproduce the custom-image case, then determine whether the expected behavior is to download the source archive or warn when the download is skipped; verify the chosen behavior with the relevant training-container tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- devops, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100