aws / aws/sagemaker-tensorflow-extensions
Accommodating more channels in sagemaker PIPE mode
- Dominant language
- C++
- Stars
- 54
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
I am training a classification model using AWS Sagemaker -TensorFlow. My training dataset is huge and distributed in 4 folders in the same AWS s3 bucket.
I defined input channels like this
inputs = {
'train1' : folder1,
'train2' : folder2,
'train3': folder3,
'train4':folder4,
'valid':folder
}
I am passing these channels 'ids' into my main train code and then reading the data using PIPE mode like this,
all_data = []
if mode = train:
for id in ids:
data = PipeModeDataset(channel=id, format = 'TFRecord')
data = parsing data here
all_data.append(data)
Now I am using _all_data_ as my whole data and doing augmentation in it and then passing it to the training script.
I got an error while doing this, (error related to data). sometimes training hangs.
What I want to know is the correct way of using multiple channels for single training using PIPE mode
Thanks
Contributor guide
Research direction
Start with the training script's PipeModeDataset usage and the AWS SageMaker input-channel setup shown in the issue. Reproduce the four-channel PIPE-mode run, inspect the reported data error and hangs, and establish the supported multi-channel behavior as the done condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, tensorflow
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100