HumanSignal / HumanSignal/label-studio
SAM2 interactive preannotation UI does not call interactive-annotating
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
### Describe the bug
Interactive SAM2 image segmentation does not work from the Label Studio UI even when the ML backend is connected, healthy, and returns valid masks when called directly.
The backend is not the failing part. The Label Studio frontend does not reliably enter interactive preannotation mode and does not fire the expected interactive request from the smart keypoint/rectangle tools.
### Reproduction
1. Run Label Studio 1.23.0.
2. Run the `segment_anything_2_image` ML backend from `label-studio-ml-backend`.
3. Configure the project with `BrushLabels`, `KeyPointLabels smart="true"`, and `RectangleLabels smart="true"`.
4. Connect the ML backend and confirm it is marked `is_interactive=true`.
5. Open a task in the normal Label Studio UI.
6. Use the smart keypoint or smart rectangle tool on the image.
### Expected behavior
After drawing a smart point/rectangle, the frontend should call:
```text
POST /api/ml/{id}/interactive-annotating
```
with a body containing:
```json
{
"task": 1,
"context": {
"result": [...]
}
}
```
Then the returned `brushlabels` RLE mask should be shown as a suggestion/annotation.
### Actual behavior
The normal UI path sends only the non-interactive prediction/test request with `context: null`, for example:
```json
{
"params": {
"login": null,
"password": null,
"context": null
}
}
```
The SAM2 backend correctly returns empty predictions for that request:
```json
{"results":[]}
```
Label Studio then logs:
```text
ML backend returns an incorrect response, results field must be a list with at least one item
```
This error is misleading in this case because the backend was called without interactive context.
### Backend verification
The same backend works when called through the interactive endpoint or directly with `params.context.result`.
Verified manually:
- `POST /api/ml/{id}/interactive-annotating` returns `200`
- response contains one `brushlabels` result
- result contains valid `value.format = "rle"`
- result contains a non-empty `value.rle`
A minimal standalone frontend that directly calls `/api/ml/{id}/interactive-annotating` works with the same project, same task, same image, same backend, and same label config.
### Suspected frontend cause
From the packaged source maps in Label Studio 1.23.0:
- `main.js.map`
- `./src/pages/DataManager/DataManager.jsx`
- `167.js.map`
- `../../libs/datamanager/src/sdk/dm-sdk.js`
- `../../libs/datamanager/src/sdk/lsf-sdk.js`
- `358.js.map`
- editor smart tool/event gating
The Data Manager finds an interactive backend and enables the `autoAnnotation` interface, but the LSF wrapper is created without `isInteractivePreannotations`.
In `../../libs/datamanager/src/sdk/dm-sdk.js`, `initLSF()` creates:
```js
new LSFWrapper(this, element, {
...this.labelStudioOptions,
task: this.store.taskStore.selected,
preload: this.preload,
isLabelStream: this.mode === "labelstream",
});
```
But `../../libs/datamanager/src/sdk/lsf-sdk.js` only enables forced auto annotation if it receives:
```js
isInteractivePreannotations: true
```
which then becomes:
```js
forceAutoAnnotation: this.isInteractivePreannotations,
forceAutoAcceptSuggestions: this.isInteractivePreannotations,
```
Without that flag, the smart tool path does not consistently emit `regionFinishedDrawing`, and the Data Manager listener never calls `/interactive-annotating`.
### Local workaround that fixed it
Patching the packaged `167.js` so `initLSF()` passes the flag fixes the issue:
```js
isInteractivePreannotations: this.hasInterface("autoAnnotation")
```
I also had to cache-bust lazy chunks because the browser can keep stale `167.js`.
### Version checked
Environment:
- Label Studio 1.23.0
- frontend commit reported by `/api/version`: `2a9bfb`
- Community edition
- Browser: Chromium/Brave on Linux
- ML backend: `segment_anything_2_image` SAM2 image backend
I inspected the published wheels from 1.13.0 through 1.23.0 and saw the same missing bridge shape in the bundled frontend code, so downgrading one version to 1.22.0 does not appear to fix this.
### Related
Possibly related to reports where SAM2 interactive masks do not appear and only empty prediction responses are seen.
Contributor guide
Assessment
This issue has not been assessed yet.