aws / aws/sagemaker-tensorflow-extensions

Accommodating more channels in sagemaker PIPE mode

Open
#135 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
54
Forks
39
PR merge metrics
No merged PRs in 30d

Description

I am training a classification model using AWS Sagemaker -TensorFlow. My training dataset is huge and distributed in 4 folders in the same AWS s3 bucket.

I defined input channels like this
inputs = {
'train1' : folder1,
'train2' : folder2,
'train3': folder3,
'train4':folder4,

'valid':folder
}

I am passing these channels 'ids' into my main train code and then reading the data using PIPE mode like this,
all_data = []
if mode = train:
for id in ids:
data = PipeModeDataset(channel=id, format = 'TFRecord')
data = parsing data here
all_data.append(data)

Now I am using _all_data_ as my whole data and doing augmentation in it and then passing it to the training script.
I got an error while doing this, (error related to data). sometimes training hangs.

What I want to know is the correct way of using multiple channels for single training using PIPE mode

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the training script's PipeModeDataset usage and the AWS SageMaker input-channel setup shown in the issue. Reproduce the four-channel PIPE-mode run, inspect the reported data error and hangs, and establish the supported multi-channel behavior as the done condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.