aws / aws/sagemaker-python-sdk

Ability to ignore specific files/folders in ModelTrainer's script mode

Open
#5,091 4 comments 0 reactions 1 assignee Claimed by @mollyheamazon View on GitHub
component: pysdk-team component: training type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**
I do not want having `.git`, `.env`, `.vscode`, `data`, `__pycache__` or any irrelevant files/folders to be uploaded to S3 artifacts when I use script mode of `ModelTrainer` in [SourceCode](https://sagemaker.readthedocs.io/en/stable/api/training/model_trainer.html#sagemaker.modules.configs.SourceCode). Moreover, coping `source_dir` may be time-consuming due to the large number of files, such as `.git` or/and `.env`.

**How would this feature be used? Please describe.**
During development/sanity-checing on local machine I have some unnecessary files/folders. The idea is to not upload them during using a script mode.
Let's say I have project structure:
```
tree
.
├── README.md
├── .git
├── pipeline1
│ ├── train.py
│ ├── data
│ ├── .env
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
├── pipeline2
│ ├── train.py
│ ├── data
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
```

```
from sagemaker.modules.train import ModelTrainer
from sagemaker.modules.configs import SourceCode, Compute

image = ""

source_code = SourceCode(
source_dir="pipeline1",
command="python train.py"
)

# or
# source_code = SourceCode(
# source_dir=".",
# command="python -m pipeline1.train")

compute = Compute(
instance_count=1,
instance_type="ml.g5.8xlarge"
)

model_trainer = ModelTrainer(
training_image=image,
source_code=source_code,
compute=compute,
)
model_trainer.train()
```
expected result:
``s3:////input/code/`` w/o ignored files/folders

**Describe alternatives you've considered**
1) Create TempDir, copy script w/o unnecessary files and pass it as source_dir.
2) Re-design the project structure to have unwanted files/dirs outside the script you want to upload. (
3) Always use BYOC instead `script` mode (isn't practical for some cases, e.g. sanity-check)

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.