aws / aws/amazon-sagemaker-examples
How to limit the upload with AWS Sagemaker? .sageignore file?
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
We have a python project set up in a rather standard way, with some modules and one module for the shared functionality. This allows you to write `import common.methods` in the file `train.py`.
```
main-folder/
.git/...
.idea/...
common-functions/
methods.py
module-one/
tests/
test_train.py
data/some_data.csv
train.py
notebook.ipynb
module-two/
....py
.gitignore
```
With AWS Sagemaker we can run our local code easily. We want to do this from the notebook `notebook.ipynb`, which is in a certain module. The way to make this work is easy, you just indicate to sagemaker to upload the source directory one up. This
```
estimator = TensorFlow(
entry_point='module-one/train.py', # the file with the imports from common
source_dir='..', # this will upload the main directory
)
```
However, now say I do not want to upload some files - for example we should not upload the `test` folders which contain some test data or the `.git` folder containing the source history. Uploading these files makes sagemaker slow (which is annoying if your on a hotspot etc.). How could we do that, while not having to divert from standard folder structure? Could we imagine a `.sageignore` file?
Contributor guide
Assessment
This issue has not been assessed yet.