aws / aws/sagemaker-python-sdk

Ability to ignore specific files/folders in ModelTrainer's script mode

Đang mở
#5,091 4 bình luận 0 reaction 1 người được giao Được @mollyheamazon nhận Xem trên GitHub
component: pysdk-team component: training type: feature request
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**Describe the feature you'd like**
I do not want having `.git`, `.env`, `.vscode`, `data`, `__pycache__` or any irrelevant files/folders to be uploaded to S3 artifacts when I use script mode of `ModelTrainer` in [SourceCode](https://sagemaker.readthedocs.io/en/stable/api/training/model_trainer.html#sagemaker.modules.configs.SourceCode). Moreover, coping `source_dir` may be time-consuming due to the large number of files, such as `.git` or/and `.env`.

**How would this feature be used? Please describe.**
During development/sanity-checing on local machine I have some unnecessary files/folders. The idea is to not upload them during using a script mode.
Let's say I have project structure:
```
tree
.
├── README.md
├── .git
├── pipeline1
│ ├── train.py
│ ├── data
│ ├── .env
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
├── pipeline2
│ ├── train.py
│ ├── data
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
```

```
from sagemaker.modules.train import ModelTrainer
from sagemaker.modules.configs import SourceCode, Compute

image = ""

source_code = SourceCode(
source_dir="pipeline1",
command="python train.py"
)

# or
# source_code = SourceCode(
# source_dir=".",
# command="python -m pipeline1.train")

compute = Compute(
instance_count=1,
instance_type="ml.g5.8xlarge"
)

model_trainer = ModelTrainer(
training_image=image,
source_code=source_code,
compute=compute,
)
model_trainer.train()
```
expected result:
``s3:////input/code/`` w/o ignored files/folders

**Describe alternatives you've considered**
1) Create TempDir, copy script w/o unnecessary files and pass it as source_dir.
2) Re-design the project structure to have unwanted files/dirs outside the script you want to upload. (
3) Always use BYOC instead `script` mode (isn't practical for some cases, e.g. sanity-check)

**Additional context**
Add any other context or screenshots about the feature request here.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.