aws / aws/amazon-sagemaker-examples

[Bug Report] JumpStart OD does not work, no correct inference performed

Open
#3,852 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[JumpStart OD Link.](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart_object_detection/Amazon_JumpStart_Object_Detection.ipynb)

**Describe the bug**
Hi there! nice to meet you!

Well, after following the jumpstart example on the link, when performing the final inference for fine-tuning (final cells of the 3.5 Section), it does not detect anything at all. Here's the print of the normalized boxes and the rectangle that is supposed to be drawn over the picture:
`normalized boxes: [[0.949622095, 0.00991547853, 1.0, 0.069772549], [0.899303257, 0.0198470056, 1.0, 0.139609843]]
classes names: [1, 1]
confidences: [1.72767649e-08, 1.73630195e-08]
Rectangle(xy=(530.839, 5.3147), width=28.1612, height=32.0834, angle=0)
Rectangle(xy=(502.711, 10.638), width=56.2895, height=64.1929, angle=0)`

if we plot them on the image, we get

![imagen](https://user-images.githubusercontent.com/71350562/225203741-85ff3811-b38d-4bea-8edf-cf4da7622df9.png)

As you can see, there is in fact, no detection at all.

Also, during training, if you look closely to the logs, you will find that it was interrupted:

(...)
[Epoch 0], Speed: 0.296 samples/sec, loss=942796.33203125.
[Epoch 0], Val_localization=241507.391, Val_classification=0.000
[Epoch 1], Speed: 5.365 samples/sec, loss=939854.900390625.
[Epoch 1], Val_localization=240784.734, Val_classification=0.000
[Epoch 2], Speed: 5.356 samples/sec, loss=937112.001953125.
[Epoch 2], Val_localization=240073.062, Val_classification=0.000
[Epoch 3], Speed: 5.325 samples/sec, loss=934391.232421875.
[Epoch 3], Val_localization=239364.844, Val_classification=0.000
[Epoch 4], Speed: 5.374 samples/sec, loss=931680.609375.
[Epoch 4], Val_localization=238658.516, Val_classification=0.000
[Epoch 5], Speed: 5.421 samples/sec, loss=928975.427734375.
[Epoch 5], Val_localization=237953.344, Val_classification=0.000
[Epoch 6], Speed: 5.393 samples/sec, loss=926273.7890625.
[Epoch 6], Val_localization=237248.953, Val_classification=0.000
[Epoch 7], Speed: 5.439 samples/sec, loss=923574.8642578125.
[Epoch 7], Val_localization=236545.172, Val_classification=0.000
[Epoch 8], Speed: 5.451 samples/sec, loss=920877.818359375.
[Epoch 8], Val_localization=235841.781, Val_classification=0.000
[Epoch 9], Speed: 5.413 samples/sec, loss=918182.267578125.
[Epoch 9], Val_localization=235138.766, Val_classification=0.000
**init
reset context of device 0 ---> HERE**
INFO:tensorflow:Writing pipeline config file to /opt/ml/model/pipeline.config
Writing pipeline config file to /opt/ml/model/pipeline.config
(...)

Other thing that draws my attention is that the validation_localization error is HUGE... so what's really happening here?

**To reproduce**
Follow the JumpStart Object Detection example

** annex **

I attach the notebook (copied directly from sagemaker examples, but with the logs included, so you can check withouth running it at all)
[sagemaker_notebook.zip](https://github.com/aws/amazon-sagemaker-examples/files/10975912/sagemaker_notebook.zip)

Contributor guide

Open the contributing guide

Research direction

Open Amazon_JumpStart_Object_Detection.ipynb and begin with the final inference cells in Section 3.5, then reproduce the reported result and review the attached notebook and training logs. Compare the training interruption, validation metrics, normalized boxes, and rendered image; done means the example completes training and produces correct detections during final inference.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, machine-learning
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.