aws / aws/amazon-sagemaker-examples
incorrect recordio format using Augmented manifest file from sagemaker ground truth
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I have used Sagemaker Ground truth to create annotation file.
Following below 2 links, I am trying to create a training job, then perform hyper parameter tuning.
- https://docs.aws.amazon.com/sagemaker/latest/dg/augmented-manifest.html
- https://github.com/awslabs/amazon-sagemaker-examples/blob/master/ground_truth_labeling_jobs/object_detection_augmented_manifest_training/object_detection_augmented_manifest_training.ipynb
However, I keep having below error
> ClientError: Unable to parse record. Please make sure input data is in correct recordio format. , exit code: 2
This is the code
```
#model config
od_model = sagemaker.estimator.Estimator(training_image,
role,
instance_count=1,
instance_type='ml.p3.2xlarge',
train_volume_size=50,
train_max_run=360000,
input_mode='Pipe',
output_path=s3_output_location,
sagemaker_session=sagemaker_session)
# Set hyperparameters
od_model.set_hyperparameters(base_network='resnet-50',
use_pretrained_model=1,
num_classes=2,
mini_batch_size=5,
epochs=30,
learning_rate=0.001,
lr_scheduler_step='10,20',
lr_scheduler_factor=0.1,
optimizer='adam',
num_training_samples=str(num_training_samples))
#data
train_data = "s3://bucket/.../manifests/output/output.manifest"
validation_data = "s3://bucket/.../manifests/output/.manifest"
train_channel = sagemaker.inputs.TrainingInput(train_data, distribution='FullyReplicated', content_type='application/x-recordio',
s3_data_type='AugmentedManifestFile',
attribute_names=attribute_names,
input_mode='Pipe',
#record_wrapping='RecordIO')
#did same for validation
```
Contributor guide
Research direction
Start with the object_detection_augmented_manifest_training/object_detection_augmented_manifest_training.ipynb example and the linked SageMaker augmented-manifest documentation. Compare its TrainingInput configuration with the issue's training and validation channels, then reproduce the training job to isolate the record parsing failure. Done means the augmented manifest trains without the record-format error and supports the stated tuning workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100