deepjavalibrary / deepjavalibrary/djl

SingleShotDetectionTranslator returns pixel coordinates instead of ratios for MXNet zoo SSD models in 0.37.0

Open
#3,889 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

## Description

After #3841 (released in 0.37.0), `SingleShotDetectionTranslator.processOutput` only divides the
box coordinates by `width`/`height` when `applyRatio` is true:

```java
// 0.36.0
double x = width > 0 ? box[0] / width : box[0];
double y = height > 0 ? box[1] / height : box[1];
double w = width > 0 ? box[2] / width - x : box[2] - x;
double h = height > 0 ? box[3] / height - y : box[3] - y;

// 0.37.0
double x = box[0];
double y = box[1];
double w = box[2] - x;
double h = box[3] - y;
```

The MXNet model-zoo SSD models emit boxes in **pixel coordinates of the resized input**
(512x512 for `ssd_512_resnet50_v1_voc`), and their zoo metadata sets `rescale: true` — but
`ObjectDetectionTranslator.ObjectDetectionBuilder.configPostProcess` only reads `applyRatio` /
`optApplyRatio`, so `applyRatio` stays `false` for these artifacts:

https://mlrepo.djl.ai/model/cv/object_detection/ai/djl/mxnet/ssd/metadata.json

```json
"arguments": {
"width": 512, "height": 512, "resize": true, "rescale": true, "threshold": 0.2,
"synsetFileName": "classes.txt",
"translatorFactory": "ai.djl.modality.cv.translator.SingleShotDetectionTranslatorFactory"
}
```

The net effect: `ai.djl.mxnet:ssd:0.0.1` returned normalized 0..1 bounds in 0.36.0 and returns
raw 0..512 pixel bounds in 0.37.0.

### Expected Behavior

`DetectedObjects` bounding boxes are ratios of the image size. That is what
`ObjectDetectionBuilder.optApplyRatio` documents:

> DetectedObject value should always bring a ratio based on the width/height instead of actual
> width/height.

and it is what the rest of DJL assumes — `BufferedImageFactory.drawBoundingBoxes` scales them
back up itself:

```java
int x = (int) (rectangle.getX() * imageWidth);
int y = (int) (rectangle.getY() * imageHeight);
g.drawRect(x, y,
(int) (rectangle.getWidth() * imageWidth),
(int) (rectangle.getHeight() * imageHeight));
```

So for the MXNet zoo SSD models 0.37.0 now breaks that contract: drawing the boxes, or cropping
with `Image#getSubImage`, puts them far outside the image.

I appreciate that #3841 is fixing a genuine double-normalization for models that already emit
ratios. The problem is that the switch is keyed on `applyRatio`, which the published MXNet SSD
metadata does not set, so those models silently changed convention. Two options that would both
work, as far as I can tell:

- add `"applyRatio": true` to the MXNet SSD artifacts' metadata in the model zoo, or
- key the behavior on something the metadata actually carries (`rescale`), or restore the
divide-by-width/height default for `SingleShotDetectionTranslator`.

`ai.djl.pytorch:ssd:0.0.1` is unaffected — it uses `PtSsdTranslatorFactory`, a different translator.

### Error Message

We hit this downstream in Apache Camel, where a route crops each detected person out of the
source image and feeds it to pose estimation. Under 0.37.0 the crop coordinates are ~512x too
large, `Image#getSubImage` fails, and no result is produced. The failing CI job is here:

https://github.com/apache/camel/actions/runs/34480900780/job/102883145412

```
[ERROR] org.apache.camel.component.djl.CvPoseEstimationTest.testDJL
[ERROR] Run 1: CvPoseEstimationTest.testDJL:37 mock://result Received message count 0, expected at least 1
[ERROR] Run 2: CvPoseEstimationTest.testDJL:37 mock://result Received message count 0, expected at least 1
[ERROR] Run 3: CvPoseEstimationTest.testDJL:37 mock://result Received message count 0, expected at least 1
```

The same test passes on 0.36.0 and the version bump is the only change in that PR
(https://github.com/apache/camel/pull/26235). I have not captured the `getSubImage` stack trace
itself — that test's route log is not archived by our CI, and the models are x86-only so I could
not rerun it locally — so the coordinate analysis above is from reading the 0.36.0/0.37.0 diff
and the zoo metadata rather than from a printed stack trace. A direct DJL-level check is below.

## How to Reproduce?

### Steps to reproduce

Load the MXNet zoo SSD and print the raw bounds on 0.36.0 and on 0.37.0:

```java
Criteria criteria = Criteria.builder()
.optApplication(Application.CV.OBJECT_DETECTION)
.setTypes(Image.class, DetectedObjects.class)
.optArtifactId("ai.djl.mxnet:ssd:0.0.1")
.build();

try (ZooModel model = criteria.loadModel();
Predictor predictor = model.newPredictor()) {
Image img = ImageFactory.getInstance().fromUrl("https://resources.djl.ai/images/pose_soccer.png");
for (DetectedObjects.DetectedObject o : predictor.predict(img). items()) {
System.out.println(o.getClassName() + " " + o.getBoundingBox().getBounds());
}
}
```

Expected on 0.36.0: bounds in 0..1.
Observed on 0.37.0: bounds in 0..512.

## What have you tried to solve it?

1. Confirmed the only delta is the DJL version (Camel PR base is otherwise unchanged `main`, and
the failure reproduced on all three surefire reruns).
2. Traced it to #3841 plus the `applyRatio`-vs-`rescale` mismatch in the MXNet SSD zoo metadata.
3. Setting `applyRatio` from the client side is not a good general workaround for us — we would
then double-normalize every model whose translator already emits ratios.

We are holding the 0.36.0 -> 0.37.0 bump on our side until this is resolved.

## Environment Info

DJL 0.37.0 (vs 0.36.0), `ai.djl.mxnet:mxnet-engine` + `ai.djl.mxnet:mxnet-model-zoo`, JDK 17,
Ubuntu 24.04 x86_64 (GitHub Actions). I have not run `./gradlew debugEnv` — the finding is from
the 0.36.0/0.37.0 source diff and the published model-zoo metadata, both linked above.

---
_Reported by Claus Ibsen (@davsclaus), Apache Camel PMC. Analysis drafted with Claude Code on his behalf._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.