aws / aws/sagemaker-python-sdk

ScriptProcessor does not check local_code config before uploading code to S3

Đang mở
#3,560 4 bình luận 0 reaction 1 người được giao Được @nargokul nhận Xem trên GitHub
component: local mode component: processing type: bug
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**Describe the bug**
When a `LocalSession` or `LocalPipelineSession` is configured to use local code, as follows
```python
session.config = {'local': {'local_code': True}}
```
the code passed to a pipeline `ProcessingStep` or directly to the `run` method of a processor (`ScriptProcessor`, `FrameworkProcessor`, ...) should not be uploaded to S3.

However, `ScriptProcessor` does not honor this. Its `_include_code_in_inputs` method (which is called unconditionally by the `_normalize_args` of the base class `Processor`, which in turn is called both when running directly and through a pipeline) unconditionally tries to upload the code to S3. https://github.com/aws/sagemaker-python-sdk/blob/554952eac259979dc714a1a9002653ced342b876/src/sagemaker/processing.py#L625

Compare this to the `Model` class, used for example in the `TrainingStep`. Its `_upload_code` method checks the session configuration and does not upload to S3 when local code is enabled. https://github.com/aws/sagemaker-python-sdk/blob/554952eac259979dc714a1a9002653ced342b876/src/sagemaker/model.py#L532

**To reproduce**
In the absence of any AWS credentials (which should not be needed when running completely locally), the following code will fail to upload the `processing.py` script to S3 (`botocore.exceptions.NoCredentialsError`). Note that, in addition to the following code, a `processing.py` file must exist in the working directory (but its contents don't matter).

Code

```python3
import boto3
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import LocalPipelineSession
from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
from sagemaker.workflow.steps import ProcessingStep

role = 'arn:aws:iam::123456789012:role/MyRole'

local_pipeline_session = LocalPipelineSession(boto_session = boto3.Session(region_name = 'eu-west-1'))
local_pipeline_session.config = {'local': {'local_code': True}}

script_processor = ScriptProcessor(
image_uri = 'docker.io/library/python:3.8',
command = ['python'],
instance_type = 'local',
instance_count = 1,
sagemaker_session = local_pipeline_session,
role = role,
)

processing_step = ProcessingStep(
name = 'Processing Step',
processor = script_processor,
code = 'processing.py',
inputs = [
ProcessingInput(
source = './input-data',
destination = '/opt/ml/processing/input',
)
],
outputs = [
ProcessingOutput(
source = '/opt/ml/processing/output',
destination = './output-data',
)
],
)

pipeline = Pipeline(
name = 'MyPipeline',
steps = [processing_step],
sagemaker_session = local_pipeline_session
)

pipeline.upsert(role_arn = role)

pipeline_run = pipeline.start()
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.126.0

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.