aws / aws/sagemaker-training-toolkit

Sagemaker Fails to download code from S3

Open
#85 0 comments 0 reactions 0 assignees View on GitHub
type: question
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

**Describe the bug**
When using sagemaker in script mode, if there are files in the `/opt/ml/code` directory, sagemaker doesn't download the packaged code from S3.

In my Dockerfile to build the image, I copy over some files into `/opt/ml/code` and I would like to use this container in a way where I can execute my scripts against it with sagemaker script mode. However it seems as though the scripts which are uploaded and packaged to s3 are never actually downloaded once the instance spins up and sagemaker starts. It fails to find the script I supplied.

When I tried running it without copying my code into `/opt/ml/code` in the Dockerfile everything worked fine. I think there is a problematic line of code which enforces this logic to only download the code when the `/opt/ml/code` directory is empty.

You can find the problematic line here:
When `run` is called and the files are attempted to be downloaded:
https://github.com/aws/sagemaker-training-toolkit/blob/7d9883cd5723bcc4d299b88e1e457b3565a7ce3a/src/sagemaker_training/entry_point.py#L92

https://github.com/aws/sagemaker-training-toolkit/blob/7d9883cd5723bcc4d299b88e1e457b3565a7ce3a/src/sagemaker_training/files.py#L127

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.