aws / aws/sagemaker-python-sdk

Something like a .sagemakerignore file option to allow for skipping of certain directories when creating source tar

Open
#4,187 4 comments 10 reactions 0 assignees View on GitHub
component: pysdk-team type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**
Similar to how in Docker there is a .dockerignore file which works like a .gitignore file, It would be great if there was a .sagemakerignore file so that any directories listed in that file would be skipped when making the tar file. I believe it would be a simple change here: https://github.com/aws/sagemaker-python-sdk/blob/744724b01bcdc3f451c9b73c34c947e12d6a8e2a/src/sagemaker/fw_utils.py#L486
Basically adding a few lines that check for the existence of a .sagemakerignore file in the directory of the `directory` variable, if it exists, and then skipping anything listed out in that file.

**How would this feature be used? Please describe.**
This would be useful to reduce the size of the source tar file which would allow a person to speed up sagemaker build creation without the need to restructure their code, if they have big files in the source directory that they don't commit to other things like git and docker, so they don't want it put into the sagemaker tarball

**Describe alternatives you've considered**
An alternative is not doing it

**Additional context**
I don't mind writing the code to do this but wanted to check in that this was something that would be accepted if I make a PR for it?

Contributor guide

Open the contributing guide

Research direction

Start in src/sagemaker/fw_utils.py around line 486, where the source tar is created, and inspect how the directory contents are selected. Confirm the expected .sagemakerignore behavior from the issue, then verify that listed directories are omitted from the resulting tar while other source files remain included.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.