Discussion: RankSEG as metric-aware post-processing for SAM-family masks

[](https://github.com/rankseg/rankseg) [](https://github.com/rankseg/rankseg/blob/main/rankseg/integration/sam.py) [](https://colab.research.google.com/github/rankseg/rankseg/blob/main/notebooks/rankseg_with_sam_family.ipynb)

Hi SAM maintainers, thank you for releasing and maintaining the SAM model family. I wanted to share a small external RankSEG integration for SAM-family mask outputs in case this direction is useful for future discussion around the final mask post-processing step.

RankSEG is not a SAM model change, a training recipe, or a boundary-refinement module. The use case here is narrower: after SAM mask logits have been restored to image-space probability maps, RankSEG replaces the final fixed-threshold binarization step with a metric-aware conversion step for Dice/IoU-style objectives.

I ran a smoke test with `facebook/sam-vit-base` on the small Hugging Face dataset `Textovic/lions-instance-segmentation`. The script streams samples through `datasets`, uses each ground-truth mask only to derive a box prompt and evaluation target, and compares the official restored-mask thresholding path against RankSEG on the same restored mask probabilities. This is not intended as a full SAM benchmark; it is only meant to make the post-processing replacement point concrete and reproducible.

| Evaluation set | Model | Prompt | Samples | Metric | Official threshold | RankSEG-Dice | RankSEG-IoU | Best delta |
| --- | --- | --- | ---: | --- | ---: | ---: | ---: | ---: |
| Full smoke test | `facebook/sam-vit-base` | GT box | 60 | Dice | 0.8445 | **0.8661** | **0.8661** | **+0.0216** |
| Full smoke test | `facebook/sam-vit-base` | GT box | 60 | IoU | 0.7766 | **0.7936** | **0.7935** | **+0.0169** |
| Selected visual examples below | `facebook/sam-vit-base` | GT box | 4 | Dice | 0.8670 | **0.9088** | **0.9088** | **+0.0419** |
| Selected visual examples below | `facebook/sam-vit-base` | GT box | 4 | IoU | 0.7734 | **0.8364** | **0.8364** | **+0.0629** |

Each panel is arranged as: ground-truth overlay plus box prompt, official thresholding, RankSEG optimized for Dice, and RankSEG optimized for IoU.

<sub>sample 0002: Dice +0.1013, IoU +0.1364




sample 0007: Dice +0.0344, IoU +0.0616






sample 0028: Dice +0.0131, IoU +0.0227




sample 0046: Dice +0.0186, IoU +0.0309</sub>

The external RankSEG repository currently includes explicit adapters for SAM-family outputs: `Sam1` for SAM1/SAM-HQ prompt masks, `Sam2` for SAM2 prompt masks, and `Sam3` for SAM3 instance/semantic masks. The adapter keeps the SAM geometry restoration explicit before RankSEG is called, so the model forward pass and prompt handling stay unchanged.

```python
from rankseg.integration.sam import Sam1

# Official SAM inference stays unchanged.
outputs = model(**inputs, multimask_output=False)

# Official baseline: restore masks and apply the usual fixed threshold.
official_masks = processor.post_process_masks(
outputs.pred_masks,
original_sizes=inputs["original_sizes"],
reshaped_input_sizes=inputs["reshaped_input_sizes"],
mask_threshold=0.0,
binarize=True,
)

# RankSEG: use the same SAM outputs and geometry metadata, then convert restored probabilities with a metric-aware objective.
rankseg_masks = Sam1(rankseg_kwargs={"metric": "dice", "solver": "RMA"}).postprocess(
outputs,
original_sizes=inputs["original_sizes"],
reshaped_input_sizes=inputs["reshaped_input_sizes"],
)
```

Related links:

- RankSEG repository: https://github.com/rankseg/rankseg
- SAM-family integration source: https://github.com/rankseg/rankseg/blob/main/rankseg/integration/sam.py
- Colab tutorial notebook: https://colab.research.google.com/github/rankseg/rankseg/blob/main/notebooks/rankseg_with_sam_family.ipynb
- RankSEG JMLR paper: https://www.jmlr.org/papers/v24/22-0712.html
- RankSEG-RMA paper: https://arxiv.org/abs/2510.15362

I do not want to assume this belongs inside the SAM repositories. My question is more modest: do maintainers think this kind of metric-aware final mask conversion is a useful external reference or future benchmark discussion point for SAM-family outputs?

I would also like to thank @statmlben and @ZixunWang for the RankSEG method. Any mistakes in this issue draft or the small smoke test setup are mine.

Start by reading the linked RankSEG integration source at rankseg/integration/sam.py and the notebooks/rankseg_with_sam_family.ipynb example, then review how SAM3 outputs are represented in this repository. The issue names no SAM3 file, test, or requested code change; completion would require a maintainer decision about accepting an external reference or defining a future benchmark.

Written by the indexing model from the issue text.