aws / aws/sagemaker-training-toolkit
Sagemaker Fails to download code from S3
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
When using sagemaker in script mode, if there are files in the `/opt/ml/code` directory, sagemaker doesn't download the packaged code from S3.
In my Dockerfile to build the image, I copy over some files into `/opt/ml/code` and I would like to use this container in a way where I can execute my scripts against it with sagemaker script mode. However it seems as though the scripts which are uploaded and packaged to s3 are never actually downloaded once the instance spins up and sagemaker starts. It fails to find the script I supplied.
When I tried running it without copying my code into `/opt/ml/code` in the Dockerfile everything worked fine. I think there is a problematic line of code which enforces this logic to only download the code when the `/opt/ml/code` directory is empty.
You can find the problematic line here:
When `run` is called and the files are attempted to be downloaded:
https://github.com/aws/sagemaker-training-toolkit/blob/7d9883cd5723bcc4d299b88e1e457b3565a7ce3a/src/sagemaker_training/entry_point.py#L92
https://github.com/aws/sagemaker-training-toolkit/blob/7d9883cd5723bcc4d299b88e1e457b3565a7ce3a/src/sagemaker_training/files.py#L127
Contributor guide
Assessment
This issue has not been assessed yet.