HumanSignal / HumanSignal/label-studio

SAM2 interactive preannotation UI does not call interactive-annotating

Open
#9,770 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

### Describe the bug

Interactive SAM2 image segmentation does not work from the Label Studio UI even when the ML backend is connected, healthy, and returns valid masks when called directly.

The backend is not the failing part. The Label Studio frontend does not reliably enter interactive preannotation mode and does not fire the expected interactive request from the smart keypoint/rectangle tools.

### Reproduction

1. Run Label Studio 1.23.0.
2. Run the `segment_anything_2_image` ML backend from `label-studio-ml-backend`.
3. Configure the project with `BrushLabels`, `KeyPointLabels smart="true"`, and `RectangleLabels smart="true"`.
4. Connect the ML backend and confirm it is marked `is_interactive=true`.
5. Open a task in the normal Label Studio UI.
6. Use the smart keypoint or smart rectangle tool on the image.

### Expected behavior

After drawing a smart point/rectangle, the frontend should call:

```text
POST /api/ml/{id}/interactive-annotating
```

with a body containing:

```json
{
"task": 1,
"context": {
"result": [...]
}
}
```

Then the returned `brushlabels` RLE mask should be shown as a suggestion/annotation.

### Actual behavior

The normal UI path sends only the non-interactive prediction/test request with `context: null`, for example:

```json
{
"params": {
"login": null,
"password": null,
"context": null
}
}
```

The SAM2 backend correctly returns empty predictions for that request:

```json
{"results":[]}
```

Label Studio then logs:

```text
ML backend returns an incorrect response, results field must be a list with at least one item
```

This error is misleading in this case because the backend was called without interactive context.

### Backend verification

The same backend works when called through the interactive endpoint or directly with `params.context.result`.

Verified manually:

- `POST /api/ml/{id}/interactive-annotating` returns `200`
- response contains one `brushlabels` result
- result contains valid `value.format = "rle"`
- result contains a non-empty `value.rle`

A minimal standalone frontend that directly calls `/api/ml/{id}/interactive-annotating` works with the same project, same task, same image, same backend, and same label config.

### Suspected frontend cause

From the packaged source maps in Label Studio 1.23.0:

- `main.js.map`
- `./src/pages/DataManager/DataManager.jsx`
- `167.js.map`
- `../../libs/datamanager/src/sdk/dm-sdk.js`
- `../../libs/datamanager/src/sdk/lsf-sdk.js`
- `358.js.map`
- editor smart tool/event gating

The Data Manager finds an interactive backend and enables the `autoAnnotation` interface, but the LSF wrapper is created without `isInteractivePreannotations`.

In `../../libs/datamanager/src/sdk/dm-sdk.js`, `initLSF()` creates:

```js
new LSFWrapper(this, element, {
...this.labelStudioOptions,
task: this.store.taskStore.selected,
preload: this.preload,
isLabelStream: this.mode === "labelstream",
});
```

But `../../libs/datamanager/src/sdk/lsf-sdk.js` only enables forced auto annotation if it receives:

```js
isInteractivePreannotations: true
```

which then becomes:

```js
forceAutoAnnotation: this.isInteractivePreannotations,
forceAutoAcceptSuggestions: this.isInteractivePreannotations,
```

Without that flag, the smart tool path does not consistently emit `regionFinishedDrawing`, and the Data Manager listener never calls `/interactive-annotating`.

### Local workaround that fixed it

Patching the packaged `167.js` so `initLSF()` passes the flag fixes the issue:

```js
isInteractivePreannotations: this.hasInterface("autoAnnotation")
```

I also had to cache-bust lazy chunks because the browser can keep stale `167.js`.

### Version checked

Environment:

- Label Studio 1.23.0
- frontend commit reported by `/api/version`: `2a9bfb`
- Community edition
- Browser: Chromium/Brave on Linux
- ML backend: `segment_anything_2_image` SAM2 image backend

I inspected the published wheels from 1.13.0 through 1.23.0 and saw the same missing bridge shape in the bundled frontend code, so downgrading one version to 1.22.0 does not appear to fix this.

### Related

Possibly related to reports where SAM2 interactive masks do not appear and only empty prediction responses are seen.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.