aws / aws/amazon-sagemaker-examples
[Bug Report] Edge Manager notebook errors loading darknet model
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker_edge_manager/sagemaker_edge_example/sagemaker_edge_example.ipynb
**Describe the bug**
Edge agent fails to load provided darknet model (yolov3-tiny). I tried on both the AMI used in the notebook (`ami-0bc87a16c757a7f07`) and the latest version (`ami-0f9543706892e0363`). The notebook has steps for two models. The keras model works fine. But if you load the darknet model there is an unclear error. After encountering this error, further errors appear if try to list models or use the keras model that was previously working. This appears to be due to the agent trying to get a list of models before doing other operations.
> [libprotobuf ERROR /codebuild/output/src790744299/src/NeoAgentSmith/third-party/grpc/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
**To Reproduce**
1. Compile and package models per the instructions in the notebook.
2. Create certificates, roles, etc.
3. Prepare a directory of all of the demo files and upload to s3.
3. Launch an EC2 instance as instructed and connect with SSH.
4. Download the files and run the agent
aws s3 sync s3://sagemaker-artifacts-XXXXXXX/em-demo/ demo/
cd demo
chmod 755 bin/*
chmod 400 certificates/us-west-2.pem
./bin/sagemaker_edge_agent_binary -a /tmp/sagemaker_edge_agent_example.sock -c sagemaker_edge_config.json
5. Connect with another SSH session and run the client commands with keras.
cd demo
./bin/sagemaker_edge_agent_client_example LoadModel keras_model demo-keras
Model demo-keras located at /home/ubuntu/demo/keras_model loaded
LoadModel succeeded
./bin/sagemaker_edge_agent_client_example Predict demo-keras keras.bmp input_1 224 224 3
Done reading the image
Predict succeeded
Flattened RAW Output Tensor:1
9.15062e-05 ...
./bin/sagemaker_edge_agent_client_example PredictAndCapture demo-keras keras.bmp input_1 224 224 3
Done reading the image
Predict succeeded
Capture Data succeeded
Flattened RAW Output Tensor:1
9.15062e-05 ...
./bin/sagemaker_edge_agent_client_example ListModels
There are 1 models
Model 0 demo-keras
ListModels succeeded
./bin/sagemaker_edge_agent_client_example UnloadModel demo-keras
Model demo-keras has been unloaded
UnLoadModel succeeded
6. Try with darknet
./bin/sagemaker_edge_agent_client_example LoadModel darknet_model demo-darknet
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
LoadModel failed
7. Try previously working commands
./bin/sagemaker_edge_agent_client_example LoadModel keras_model demo-keras
Model demo-keras located at /home/ubuntu/demo/keras_model loaded
LoadModel succeeded
./bin/sagemaker_edge_agent_client_example ListModels
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
ListModels failed
./bin/sagemaker_edge_agent_client_example Predict demo-keras keras.bmp input_1 224 224 3
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
Model demo-keras has not been loaded
[20210430-edgeagent.txt](https://github.com/aws/amazon-sagemaker-examples/files/6408545/20210430-edgeagent.txt)
Contributor guide
Assessment
This issue has not been assessed yet.