aws / aws/amazon-sagemaker-examples
[Bug Report] JumpStart OD does not work, no correct inference performed
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[JumpStart OD Link.](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart_object_detection/Amazon_JumpStart_Object_Detection.ipynb)
**Describe the bug**
Hi there! nice to meet you!
Well, after following the jumpstart example on the link, when performing the final inference for fine-tuning (final cells of the 3.5 Section), it does not detect anything at all. Here's the print of the normalized boxes and the rectangle that is supposed to be drawn over the picture:
`normalized boxes: [[0.949622095, 0.00991547853, 1.0, 0.069772549], [0.899303257, 0.0198470056, 1.0, 0.139609843]]
classes names: [1, 1]
confidences: [1.72767649e-08, 1.73630195e-08]
Rectangle(xy=(530.839, 5.3147), width=28.1612, height=32.0834, angle=0)
Rectangle(xy=(502.711, 10.638), width=56.2895, height=64.1929, angle=0)`
if we plot them on the image, we get

As you can see, there is in fact, no detection at all.
Also, during training, if you look closely to the logs, you will find that it was interrupted:
(...)
[Epoch 0], Speed: 0.296 samples/sec, loss=942796.33203125.
[Epoch 0], Val_localization=241507.391, Val_classification=0.000
[Epoch 1], Speed: 5.365 samples/sec, loss=939854.900390625.
[Epoch 1], Val_localization=240784.734, Val_classification=0.000
[Epoch 2], Speed: 5.356 samples/sec, loss=937112.001953125.
[Epoch 2], Val_localization=240073.062, Val_classification=0.000
[Epoch 3], Speed: 5.325 samples/sec, loss=934391.232421875.
[Epoch 3], Val_localization=239364.844, Val_classification=0.000
[Epoch 4], Speed: 5.374 samples/sec, loss=931680.609375.
[Epoch 4], Val_localization=238658.516, Val_classification=0.000
[Epoch 5], Speed: 5.421 samples/sec, loss=928975.427734375.
[Epoch 5], Val_localization=237953.344, Val_classification=0.000
[Epoch 6], Speed: 5.393 samples/sec, loss=926273.7890625.
[Epoch 6], Val_localization=237248.953, Val_classification=0.000
[Epoch 7], Speed: 5.439 samples/sec, loss=923574.8642578125.
[Epoch 7], Val_localization=236545.172, Val_classification=0.000
[Epoch 8], Speed: 5.451 samples/sec, loss=920877.818359375.
[Epoch 8], Val_localization=235841.781, Val_classification=0.000
[Epoch 9], Speed: 5.413 samples/sec, loss=918182.267578125.
[Epoch 9], Val_localization=235138.766, Val_classification=0.000
**init
reset context of device 0 ---> HERE**
INFO:tensorflow:Writing pipeline config file to /opt/ml/model/pipeline.config
Writing pipeline config file to /opt/ml/model/pipeline.config
(...)
Other thing that draws my attention is that the validation_localization error is HUGE... so what's really happening here?
**To reproduce**
Follow the JumpStart Object Detection example
** annex **
I attach the notebook (copied directly from sagemaker examples, but with the logs included, so you can check withouth running it at all)
[sagemaker_notebook.zip](https://github.com/aws/amazon-sagemaker-examples/files/10975912/sagemaker_notebook.zip)
Contributor guide
Research direction
Open Amazon_JumpStart_Object_Detection.ipynb and begin with the final inference cells in Section 3.5, then reproduce the reported result and review the attached notebook and training logs. Compare the training interruption, validation metrics, normalized boxes, and rendered image; done means the example completes training and produces correct detections during final inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, machine-learning
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100