aws / aws/amazon-sagemaker-examples

[Example Request] - Distributed Processing SKLEARN : AWS Sagemaker

Open
#2,852 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I have a few raw .csv files in my S3 bucket. How can I process them in parallel to reduce run time? See comments on where I require a little help. I am using SKLearnProcessor and s3_data_distribution_type='ShardedByS3Key'

Screenshot 2021-08-02 at 16 37 30

```
%%writefile preprocessing/preprocessing_sklearn.py

import pandas as pd
import argparse
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import os

def process(input_data_path):
df = pd.read_csv(input_data_path)
# drop first col (unamed: 0)
df = df.iloc[: , 1:]

features = df.iloc[:,1:]
headers = features.columns
labels = df.iloc[:,0]

scaler = StandardScaler()

normalized_x_train = scaler.fit_transform(features)

# write
pd.DataFrame(normalized_x_train).to_csv((os.path.join('/opt/ml/processing/output/train', 'train_features.csv')), header=False, index=False)
pd.DataFrame(labels).to_csv((os.path.join('/opt/ml/processing/output/train', 'train_labels.csv')), header=False, index=False)

if __name__ == '__main__':
# HOW DO I MAKE THIS DYNAMIC? CHUNK_1.CSV, CHUNK_2.CSV ETC
input_data_path = os.path.join("/opt/ml/processing/input", "train-data-with-header.csv")
process(input_data_path)
```
My calling fn -

```
from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput
import timeit

start = timeit.default_timer()
# WHAT SHOULD BE MY SOURCE?
source = "s3://sagemaker-end-to-end/data_tuning/train/chunk_0.csv"
source2 = "s3://sagemaker-end-to-end/data_tuning/train/"

sklearn_processor = SKLearnProcessor(framework_version='0.23-1',
role=role,
instance_type='ml.m5.xlarge',
instance_count=2,
base_job_name = 'preprocess-sklearn'
)

sklearn_processor.run(
code='preprocessing/preprocessing_sklearn.py',
inputs=[
ProcessingInput(
source=source2,
s3_data_distribution_type='ShardedByS3Key',
destination='/opt/ml/processing/input')
],

outputs=[
ProcessingOutput(
source='/opt/ml/processing/output/train',
destination= make_url(store_bucket, "preprocess_sklearn", "train")
),
#
ProcessingOutput(
source='/opt/ml/processing/output/test',
destination= make_url(store_bucket, "preprocess_sklearn", "test")
)
]

)

stop = timeit.default_timer()

print('Time: ', stop - start)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.