aws / aws/sagemaker-training-toolkit

Silent Failure if custom image puts something into /opt/ml/code

Open
#222 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

Hi, I was making a new Docker image for training:

```
FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04
COPY src/requirements.txt /opt/ml/code/requirements.txt
RUN pip install --no-cache-dir -r /opt/ml/code/requirements.txt
```

And I found that when I do that, my training image could no longer find the files that usually get copied in when the container runs. I traced it back to this line, which checks if the /opt/ml/code folder exists, and if it exists at all it just skips the step that copies over the sourcedir.tar.gz file from that URI.

Should the logic be changed so that it doesn't skip downloading the file, or maybe at least it should give a warning that it's skipping the download?

https://github.com/aws/sagemaker-training-toolkit/blob/628166c157751ae2a46fddc11a7a8cac765fb22c/src/sagemaker_training/files.py#L134

Contributor guide

Open the contributing guide

Research direction

Start at src/sagemaker_training/files.py around line 134 and trace the condition that handles an existing /opt/ml/code directory and the sourcedir.tar.gz URI. Reproduce the custom-image case, then determine whether the expected behavior is to download the source archive or warn when the download is skipped; verify the chosen behavior with the relevant training-container tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
devops, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.