aws / aws/sagemaker-python-sdk

ScriptProcessor does not check local_code config before uploading code to S3

Open
#3,560 4 comments 0 reactions 1 assignee Claimed by @nargokul View on GitHub
component: local mode component: processing type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
2d 2h
Merged PRs (30d)
32

Description

**Describe the bug**
When a `LocalSession` or `LocalPipelineSession` is configured to use local code, as follows
```python
session.config = {'local': {'local_code': True}}
```
the code passed to a pipeline `ProcessingStep` or directly to the `run` method of a processor (`ScriptProcessor`, `FrameworkProcessor`, ...) should not be uploaded to S3.

However, `ScriptProcessor` does not honor this. Its `_include_code_in_inputs` method (which is called unconditionally by the `_normalize_args` of the base class `Processor`, which in turn is called both when running directly and through a pipeline) unconditionally tries to upload the code to S3. https://github.com/aws/sagemaker-python-sdk/blob/554952eac259979dc714a1a9002653ced342b876/src/sagemaker/processing.py#L625

Compare this to the `Model` class, used for example in the `TrainingStep`. Its `_upload_code` method checks the session configuration and does not upload to S3 when local code is enabled. https://github.com/aws/sagemaker-python-sdk/blob/554952eac259979dc714a1a9002653ced342b876/src/sagemaker/model.py#L532

**To reproduce**
In the absence of any AWS credentials (which should not be needed when running completely locally), the following code will fail to upload the `processing.py` script to S3 (`botocore.exceptions.NoCredentialsError`). Note that, in addition to the following code, a `processing.py` file must exist in the working directory (but its contents don't matter).

Code

```python3
import boto3
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import LocalPipelineSession
from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
from sagemaker.workflow.steps import ProcessingStep

role = 'arn:aws:iam::123456789012:role/MyRole'

local_pipeline_session = LocalPipelineSession(boto_session = boto3.Session(region_name = 'eu-west-1'))
local_pipeline_session.config = {'local': {'local_code': True}}

script_processor = ScriptProcessor(
image_uri = 'docker.io/library/python:3.8',
command = ['python'],
instance_type = 'local',
instance_count = 1,
sagemaker_session = local_pipeline_session,
role = role,
)

processing_step = ProcessingStep(
name = 'Processing Step',
processor = script_processor,
code = 'processing.py',
inputs = [
ProcessingInput(
source = './input-data',
destination = '/opt/ml/processing/input',
)
],
outputs = [
ProcessingOutput(
source = '/opt/ml/processing/output',
destination = './output-data',
)
],
)

pipeline = Pipeline(
name = 'MyPipeline',
steps = [processing_step],
sagemaker_session = local_pipeline_session
)

pipeline.upsert(role_arn = role)

pipeline_run = pipeline.start()
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.126.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.