aws / aws/amazon-sagemaker-examples

[Bug Report] Preprocessing script in ScriptProcessor executed twice

Open
#2,877 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Describe the bug**
When running the code below, the script preprocessing.py is execute twice. For instance if preprocessing.py simply consists of `print('Hello')`, then the output of script_processor.run(...) will be:
Hello
Hello

**To reproduce**
Run the code below:
```
from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
from sagemaker.dataset_definition.inputs import DatasetDefinition, RedshiftDatasetDefinition
from sagemaker import session
import boto3

boto_session = boto3.session.Session(profile_name='[my_profile_name]', region_name='eu-central-1')

script_processor = ScriptProcessor(image_uri='[my_image_uri]',
base_job_name='[my_job_name]',
role='[my_role]',
instance_type='ml.c5.4xlarge',
instance_count=1,
sagemaker_session=session.Session(boto_session=boto_session),
command=['python3'])

processing_input = ProcessingInput(
input_name='my_input_name',
app_managed=False,
dataset_definition=DatasetDefinition(
local_path='/opt/ml/processing/input/',
data_distribution_type='FullyReplicated',
input_mode='File',
redshift_dataset_definition=RedshiftDatasetDefinition(
cluster_id='[my_cluster_id]',
database='live',
db_user='[my_db_user]',
query_string='[my_sql_query]',
cluster_role_arn='[my_cluster_role_arn]',
output_s3_uri='s3://{bucket}/data/output/parquet',
output_format='PARQUET'
)
)
)

script_processor.run(code='preprocessing.py',
inputs=[processing_input],
outputs=[ProcessingOutput(source='/opt/ml/processing/output/parquet',
destination='s3://{bucket}/data/output/parquet')
],
wait=True
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.