aws / aws/amazon-sagemaker-examples
[Example Request] - Distributed Processing SKLEARN : AWS Sagemaker
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I have a few raw .csv files in my S3 bucket. How can I process them in parallel to reduce run time? See comments on where I require a little help. I am using SKLearnProcessor and s3_data_distribution_type='ShardedByS3Key'

```
%%writefile preprocessing/preprocessing_sklearn.py
import pandas as pd
import argparse
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import os
def process(input_data_path):
df = pd.read_csv(input_data_path)
# drop first col (unamed: 0)
df = df.iloc[: , 1:]
features = df.iloc[:,1:]
headers = features.columns
labels = df.iloc[:,0]
scaler = StandardScaler()
normalized_x_train = scaler.fit_transform(features)
# write
pd.DataFrame(normalized_x_train).to_csv((os.path.join('/opt/ml/processing/output/train', 'train_features.csv')), header=False, index=False)
pd.DataFrame(labels).to_csv((os.path.join('/opt/ml/processing/output/train', 'train_labels.csv')), header=False, index=False)
if __name__ == '__main__':
# HOW DO I MAKE THIS DYNAMIC? CHUNK_1.CSV, CHUNK_2.CSV ETC
input_data_path = os.path.join("/opt/ml/processing/input", "train-data-with-header.csv")
process(input_data_path)
```
My calling fn -
```
from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput
import timeit
start = timeit.default_timer()
# WHAT SHOULD BE MY SOURCE?
source = "s3://sagemaker-end-to-end/data_tuning/train/chunk_0.csv"
source2 = "s3://sagemaker-end-to-end/data_tuning/train/"
sklearn_processor = SKLearnProcessor(framework_version='0.23-1',
role=role,
instance_type='ml.m5.xlarge',
instance_count=2,
base_job_name = 'preprocess-sklearn'
)
sklearn_processor.run(
code='preprocessing/preprocessing_sklearn.py',
inputs=[
ProcessingInput(
source=source2,
s3_data_distribution_type='ShardedByS3Key',
destination='/opt/ml/processing/input')
],
outputs=[
ProcessingOutput(
source='/opt/ml/processing/output/train',
destination= make_url(store_bucket, "preprocess_sklearn", "train")
),
#
ProcessingOutput(
source='/opt/ml/processing/output/test',
destination= make_url(store_bucket, "preprocess_sklearn", "test")
)
]
)
stop = timeit.default_timer()
print('Time: ', stop - start)
```
Contributor guide
Assessment
This issue has not been assessed yet.