aws / aws/sagemaker-python-sdk
Ability to ignore specific files/folders in ModelTrainer's script mode
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the feature you'd like**
I do not want having `.git`, `.env`, `.vscode`, `data`, `__pycache__` or any irrelevant files/folders to be uploaded to S3 artifacts when I use script mode of `ModelTrainer` in [SourceCode](https://sagemaker.readthedocs.io/en/stable/api/training/model_trainer.html#sagemaker.modules.configs.SourceCode). Moreover, coping `source_dir` may be time-consuming due to the large number of files, such as `.git` or/and `.env`.
**How would this feature be used? Please describe.**
During development/sanity-checing on local machine I have some unnecessary files/folders. The idea is to not upload them during using a script mode.
Let's say I have project structure:
```
tree
.
├── README.md
├── .git
├── pipeline1
│ ├── train.py
│ ├── data
│ ├── .env
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
├── pipeline2
│ ├── train.py
│ ├── data
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
```
```
from sagemaker.modules.train import ModelTrainer
from sagemaker.modules.configs import SourceCode, Compute
image = ""
source_code = SourceCode(
source_dir="pipeline1",
command="python train.py"
)
# or
# source_code = SourceCode(
# source_dir=".",
# command="python -m pipeline1.train")
compute = Compute(
instance_count=1,
instance_type="ml.g5.8xlarge"
)
model_trainer = ModelTrainer(
training_image=image,
source_code=source_code,
compute=compute,
)
model_trainer.train()
```
expected result:
``s3:////input/code/`` w/o ignored files/folders
**Describe alternatives you've considered**
1) Create TempDir, copy script w/o unnecessary files and pass it as source_dir.
2) Re-design the project structure to have unwanted files/dirs outside the script you want to upload. (
3) Always use BYOC instead `script` mode (isn't practical for some cases, e.g. sanity-check)
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.