aws / aws/amazon-sagemaker-examples

incorrect recordio format using Augmented manifest file from sagemaker ground truth

Open
#4,056 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I have used Sagemaker Ground truth to create annotation file.

Following below 2 links, I am trying to create a training job, then perform hyper parameter tuning.

- https://docs.aws.amazon.com/sagemaker/latest/dg/augmented-manifest.html
- https://github.com/awslabs/amazon-sagemaker-examples/blob/master/ground_truth_labeling_jobs/object_detection_augmented_manifest_training/object_detection_augmented_manifest_training.ipynb

However, I keep having below error

> ClientError: Unable to parse record. Please make sure input data is in correct recordio format. , exit code: 2

This is the code
```
#model config
od_model = sagemaker.estimator.Estimator(training_image,
role,
instance_count=1,
instance_type='ml.p3.2xlarge',
train_volume_size=50,
train_max_run=360000,
input_mode='Pipe',
output_path=s3_output_location,
sagemaker_session=sagemaker_session)

# Set hyperparameters
od_model.set_hyperparameters(base_network='resnet-50',
use_pretrained_model=1,
num_classes=2,
mini_batch_size=5,
epochs=30,
learning_rate=0.001,
lr_scheduler_step='10,20',
lr_scheduler_factor=0.1,
optimizer='adam',
num_training_samples=str(num_training_samples))

#data
train_data = "s3://bucket/.../manifests/output/output.manifest"
validation_data = "s3://bucket/.../manifests/output/.manifest"

train_channel = sagemaker.inputs.TrainingInput(train_data, distribution='FullyReplicated', content_type='application/x-recordio',
s3_data_type='AugmentedManifestFile',
attribute_names=attribute_names,
input_mode='Pipe',
#record_wrapping='RecordIO')
#did same for validation

```

Contributor guide

Open the contributing guide

Research direction

Start with the object_detection_augmented_manifest_training/object_detection_augmented_manifest_training.ipynb example and the linked SageMaker augmented-manifest documentation. Compare its TrainingInput configuration with the issue's training and validation channels, then reproduce the training job to isolate the record parsing failure. Done means the augmented manifest trains without the record-format error and supports the stated tuning workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.