aws / aws/amazon-sagemaker-examples
[Bug Report] Preprocessing script in ScriptProcessor executed twice
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Describe the bug**
When running the code below, the script preprocessing.py is execute twice. For instance if preprocessing.py simply consists of `print('Hello')`, then the output of script_processor.run(...) will be:
Hello
Hello
**To reproduce**
Run the code below:
```
from sagemaker.processing import ProcessingInput, ProcessingOutput, ScriptProcessor
from sagemaker.dataset_definition.inputs import DatasetDefinition, RedshiftDatasetDefinition
from sagemaker import session
import boto3
boto_session = boto3.session.Session(profile_name='[my_profile_name]', region_name='eu-central-1')
script_processor = ScriptProcessor(image_uri='[my_image_uri]',
base_job_name='[my_job_name]',
role='[my_role]',
instance_type='ml.c5.4xlarge',
instance_count=1,
sagemaker_session=session.Session(boto_session=boto_session),
command=['python3'])
processing_input = ProcessingInput(
input_name='my_input_name',
app_managed=False,
dataset_definition=DatasetDefinition(
local_path='/opt/ml/processing/input/',
data_distribution_type='FullyReplicated',
input_mode='File',
redshift_dataset_definition=RedshiftDatasetDefinition(
cluster_id='[my_cluster_id]',
database='live',
db_user='[my_db_user]',
query_string='[my_sql_query]',
cluster_role_arn='[my_cluster_role_arn]',
output_s3_uri='s3://{bucket}/data/output/parquet',
output_format='PARQUET'
)
)
)
script_processor.run(code='preprocessing.py',
inputs=[processing_input],
outputs=[ProcessingOutput(source='/opt/ml/processing/output/parquet',
destination='s3://{bucket}/data/output/parquet')
],
wait=True
)
```
Contributor guide
Assessment
This issue has not been assessed yet.