aws / aws/sagemaker-python-sdk

Possible class or Enum for SageMaker Job

Open
#4,935 0 comments 0 reactions 1 assignee Claimed by @nargokul View on GitHub
component: processing type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**

When working with SageMaker we are often defining sources and destinations for data and artifacts within our jobs.

For instance a ProcessingInput for a processing job will be defined like:

```python
ProcessingInput(
source='s3://path/to/my/input-data.csv',
destination='/opt/ml/processing/input'
)
```

and an output would be defined like:

```python
ProcessingOutput(source='/opt/ml/processing/output/train', destination='s3://...')
```

And the `/opt/ml/...` filepaths determine where resources exist in the container and need to be correctly handled in our processing/training code.

There are other locations similar to this for training and tuning and there are environment variables that can control the default locations where resources are expected to be inside the local container.

To keep consistency across our SageMaker projects we usually end up defining a basic class or an Enum in a config file. This helps avoid things like typos and allows users to keep consistent conventions between projects.

Something like a class or Enum that define the most commonly used locations could be helpful for new users to SageMaker and prevent users from having to reference documentation (which can sometimes be a little scattered) to remember the conventional locations.

For example:

```python
class SageMakerProcessingChannels:
PROCESSING_INPUT_CHANNEL = "/opt/ml/processing/input"
PROCESSING_OUTPUT_CHANNEL = "/opt/ml/processing/output"
PROCESSING_TRAIN_OUTPUT_CHANNEL = "/opt/ml/processing/output/train"
PROCESSING_VALIDATION_OUTPUT_CHANNEL = "/opt/ml/processing/output/validation"
PROCESSING_TEST_OUTPUT_CHANNEL = "/opt/ml/processing/output/test"
PROCESSING_TEMP = "/opt/ml/processing/temp"
```

**How would this feature be used? Please describe.**
This feature would help standardize some of these common locations and provide IDE code-completion support for common
parameters when working in SageMaker.

Now our processing inputs and outputs would be:

```python
inputs = [ProcessingInput(
source='s3://path/to/my/input-data.csv',
destination=SageMakerProcessingChannels.PROCESSING_INPUT_CHANNEL
)]
outputs = [
ProcessingOutput(source=SageMakerProcesingChannels.PROCESSING_TRAIN_OUTPUT_CHANNEL, destination='s3://...')
]
```

**Describe alternatives you've considered**
We currently use a config that does this and use a cookie cutter template to initialize the SageMaker datascience projects to help promote uniformity across teams.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.