[Results]Cropped-RPI cervical rootlets: fold-0 accuracy and inference benchmark

## Context

This issue reports preliminary fold-0 results for the cropped-image training described in #105. The primary objective of #105 is to determine whether cropping speeds up cervical-rootlet inference without an unacceptable loss of segmentation quality. This report therefore separates accuracy, segmentation-only timing on precomputed crops, and the end-to-end deployment timing that still needs to be measured. It follows the reporting style of previous results issues such as #84, #7, and #5.

The objectives here are to:

- report performance on the held-out **test** images, not only the fold-0 validation set;
- document the exact checkpoint and commands used;
- compare segmentation quality with the existing uncropped model using a matched protocol;
- compare inference time with the uncropped model, including the cost of detecting, cropping, and restoring the cropped prediction to the native image grid.

## Data and split

I used the 17 images marked `Test` in [`MP2RAGE_T2w_fold_splits.csv`](https://github.com/ivadomed/model-spinal-rootlets/blob/main/training/hc-leipzig-7t-mp2rage/MP2RAGE_T2w_fold_splits.csv):

- 5 T2w acquisitions with `desc-staple` consensus references;
- 12 MP2RAGE images from four subjects (`INV1`, `INV2`, and `UNIT1` for each subject). The level-specific `UNIT1` reference is shared by the three coregistered contrasts for a subject.

This is the same 17-image test cohort used to evaluate the existing uncropped `MULTICON_v2` model in #84.

For the preliminary cropped-model inference below, each full test image was cropped using the `sc_crop` 0.6.0 detector-only default workflow and normalized to RPI orientation. The reference segmentation did not define or expand the crop. All 17 detector crops retained every reference foreground voxel, and every prediction/reference grid was verified to match.

## Relation to the existing uncropped benchmark

The current uncropped multi-contrast model has already been evaluated for segmentation quality on this test split. #84 reports per-level Dice distributions and qualitative examples for `MULTICON_v2` with the increased patch size (`352 × 96 × 128`) on 4 INV1, 4 INV2, 4 UNIT1, and 5 T2w images.

That result is useful context, but the available figure is not yet a matched numerical comparison with the cropped model: it reports levels 2–8 as box plots, does not show T1/label 9, and does not provide a machine-readable per-image table using the metric definition below. I also could not find a measured cervical-model inference time in the repository's existing results issues. Mentions of inference duration elsewhere are qualitative or concern different lumbar/CPU pipelines, so they cannot serve as the uncropped baseline for #105.

The remaining comparison is therefore to rerun the uncropped model and evaluate both pipelines on the same 17 source images, hardware, software environment, labels, metric implementation, and post-processing policy. Because the results reported here are from cropped fold 0 only, they should not be compared directly with an all-fold or ensemble result unless the fold/ensemble configurations are explicitly matched.

## Intended inference pipelines

The two models require different input handling, but must start from the same full source image:

| Arm | Pipeline | Why |
| --- | --- | --- |
| Current uncropped model | full RPI image → uncropped nnUNet → native-grid prediction | This is the model's native input distribution and the fair runtime baseline. Do **not** run `sc_crop` first for this arm. |
| Cropped model | full image → detector-only `sc_crop` → RPI cropped image → cropped nnUNet → restore prediction to the native grid | The model was trained on detector-derived crops. Full-image inference would introduce a train/inference distribution mismatch and defeat the intended speed optimization. |

The crop must be image-derived and must not use the test reference segmentation. Accuracy comparisons should be computed after the cropped prediction has been restored to the full native grid. This is consistent with the architecture linked from #105 in [SCT PR #5260](https://github.com/spinalcordtoolbox/spinalcordtoolbox/pull/5260): detect the spinal-cord bounding box, crop, run nnUNet, and restore the result. That PR does not yet implement this rootlets pipeline, but it provides the intended deployment pattern. A similar requirement is documented for a crop-trained lumbar-rootlets model in #89.

## Training setup

- Repository revision: [`4861d5f`](https://github.com/ivadomed/model-spinal-rootlets/commit/4861d5f21f62984992d5353ed843f099c2c3f882) on `ky/105-cropped-rootlets-training`
- Dataset: `Dataset402_CervicalRootletsCroppedRPI`
- Trainer/configuration: `nnUNetTrainer_2000epochs`, `3d_fullres`, fold 0
- Fold-0 split: 59 training / 16 validation images
- Patch size: `256 × 128 × 64`
- Target spacing: `0.7065 × 0.7000 × 0.7065 mm`
- Batch size: 2
- Initial learning rate: 0.01
- Weight decay: 3e-5
- Environment: Python 3.10.20, nnUNetv2 2.4.2, PyTorch 2.5.1+cu121, NVIDIA RTX A6000

Training command:

```console
CUDA_VISIBLE_DEVICES=0 nnUNetv2_train 402 3d_fullres 0 \
-tr nnUNetTrainer_2000epochs \
-device cuda
```

Training completed all 2000 epochs. The selected best checkpoint was written at epoch 1089 (`_best_ema = 0.68394107`):

```text
/home/kuanyiw/experiments/saved_models/checkpoint_best_2000epochs_rpi_0.pth
SHA256: 51354be3cb084191c866f7817eae7f7950f8eff3e161276e8396267b56a90f49
```

## Training progress

## Test inference

```console
export nnUNet_raw=/home/kuanyiw/projects/rootlets/data
export nnUNet_preprocessed=/home/kuanyiw/projects/rootlets/data/unet_output_cropped_rpi/nnUNet_preprocessed_clean_rpi
export nnUNet_results=/home/kuanyiw/projects/rootlets/data/unet_output_cropped_rpi/nnUNet_results_clean_rpi

nnUNetv2_predict \
-i /home/kuanyiw/projects/rootlets/data/canonical_test_17/cropped_rpi/imagesTs \
-o /home/kuanyiw/experiments/results/cropped_rpi_2000epochs_fold0_best/test17_predictions \
-d 402 \
-tr nnUNetTrainer_2000epochs \
-p nnUNetPlans \
-c 3d_fullres \
-f 0 \
-chk checkpoint_best.pth \
-device cuda
```

The command produced 17/17 segmentations with exit status 0. Wall time was 6 min 51.99 s for the full batch on one RTX A6000, or 24.23 s/image when the batch wall time is divided by 17. This includes nnUNet startup, preprocessing/resampling, inference, and export of the already-cropped images; it does **not** include detector cropping or mapping predictions back to the native full-image grid. It is therefore a preliminary segmentation-only batch measurement, not the end-to-end runtime required to answer #105.

Across the 17 cases, detector cropping reduced the input from 254,205,952 to 53,603,008 voxels: 78.9% fewer voxels, or a 4.74× reduction. This is a workload proxy, not a measured speedup; preprocessing, sliding-window inference, I/O, detector startup, and native-grid restoration need to be timed directly.

### Matched runtime comparison for #105

| Pipeline | Input | Model configuration | Components timed | Total for 17 | Per-case timing | Speedup |
| --- | --- | --- | --- | ---: | ---: | ---: |
| Current uncropped baseline | Full RPI source images | Released r20250318 `fold_all`, `checkpoint_final.pth` | preprocessing + nnUNet + native-grid export | **18:21.75** | **64.81 s/image amortized** | 1.00× |
| Cropped, segmentation only | Precomputed detector crops, RPI | Fold 0, `checkpoint_best.pth` | preprocessing + nnUNet + cropped-grid export | **5:38.93** | **19.94 s/image amortized** | **3.25×** |
| Cropped, end to end | Full source images | Fold 0, best checkpoint | detector + crop + preprocessing + nnUNet + native-grid restoration/export | TODO | TODO | TODO |

This first matched run used the same nnUNetv2 2.4.2 / PyTorch 2.5.1+cu121 environment, physical GPU 1 on the same otherwise-idle RTX A6000, one loaded network, TTA disabled, sliding-window step size 0.5, three preprocessing workers, and three export workers. One `sspr17 UNIT1` smoke inference was run for each model before its 17-case batch. Cropping reduced segmentation-only batch wall time by 69.2%.

Both models were invoked directly with `nnUNetv2_predict` on explicitly verified RPI inputs. This avoids an orientation ambiguity in the released model package: its `dataset.json` does not declare `image_orientation`, even though the release/training documentation specifies RPI, so the generic SCT wrapper would otherwise fall back to LPI.

The matched TTA-disabled predictions also produced the following accuracy comparison:

| Pipeline | Macro level Dice | Binary Dice | C2–C8 macro Dice |
| --- | ---: | ---: | ---: |
| Released uncropped model | 0.571 | 0.613 | 0.638 |
| Cropped fold-0 model | **0.633** | **0.654** | **0.649** |

The cropped model had higher per-image macro Dice in 16/17 cases and higher binary Dice in 17/17 cases. Mean paired differences were +0.0623 macro Dice and +0.0419 binary Dice; when T1 is excluded, the C2–C8 macro difference was +0.0111. This is a production-model comparison, not a controlled training ablation: the released checkpoint is a `fold_all` model trained on all 76 non-test images, whereas the cropped checkpoint is fold 0 (59 training / 16 validation images).

For the final timing estimate, repeat each arm at least three times while alternating their order, and report median [IQR] as well as total batch time. For the cropped end-to-end arm, record detector/crop, nnUNet, restoration/export, and total time separately. Also report peak GPU memory if available.

The relevant speedups are:

- **segmentation-only speedup:** uncropped nnUNet time / cropped nnUNet time;
- **end-to-end speedup:** uncropped total time / (detector + crop + cropped nnUNet + restoration/export).

Only the end-to-end value answers whether a user actually receives a result faster.

## Test Dice

The primary semantic metric is the per-image macro-average over spinal-level labels present in the union of prediction and reference. A class absent from both is excluded; a missed reference class scores 0. Binary Dice collapses all spinal levels to rootlet foreground. Values below are mean ± sample SD and median [IQR]. No post-processing was applied.

| Contrast | n | Macro level Dice | Macro median [IQR] | Binary Dice | Binary median [IQR] |
| --- | ---: | ---: | ---: | ---: | ---: |
| **All test images** | **17** | **0.646 ± 0.058** | **0.655 [0.627–0.690]** | **0.669 ± 0.050** | **0.687 [0.643–0.704]** |
| T2w | 5 | 0.611 ± 0.074 | 0.639 [0.594–0.655] | 0.670 ± 0.064 | 0.696 [0.643–0.719] |
| INV1 | 4 | 0.623 ± 0.049 | 0.637 [0.608–0.652] | 0.629 ± 0.047 | 0.642 [0.622–0.650] |
| INV2 | 4 | 0.685 ± 0.030 | 0.693 [0.678–0.699] | 0.694 ± 0.029 | 0.703 [0.690–0.707] |
| UNIT1 | 4 | 0.674 ± 0.044 | 0.685 [0.663–0.697] | 0.681 ± 0.041 | 0.691 [0.671–0.701] |

### Dice by spinal level

| Level | GT present | Predicted | n valid | Dice | Median [IQR] |
| --- | ---: | ---: | ---: | ---: | ---: |
| C2 | 17 | 17 | 17 | 0.647 ± 0.101 | 0.637 [0.575–0.738] |
| C3 | 17 | 17 | 17 | 0.671 ± 0.089 | 0.680 [0.612–0.730] |
| C4 | 17 | 17 | 17 | 0.639 ± 0.071 | 0.632 [0.611–0.685] |
| C5 | 17 | 17 | 17 | 0.642 ± 0.131 | 0.676 [0.629–0.709] |
| C6 | 17 | 17 | 17 | 0.703 ± 0.056 | 0.699 [0.669–0.747] |
| C7 | 17 | 17 | 17 | 0.683 ± 0.055 | 0.693 [0.649–0.716] |
| C8 | 17 | 17 | 17 | 0.651 ± 0.172 | 0.698 [0.663–0.738] |
| **T1** | **3** | **0** | **3** | **0.000 ± 0.000** | **0.000 [0.000–0.000]** |

Per-image test Dice

| Case | Contrast | Macro level Dice | Binary Dice |
| --- | --- | ---: | ---: |
| `ds004507_sub-007_ses-headNormal_T2w` | T2w | 0.491 | 0.571 |
| `ds004507_sub-010_ses-headUp_T2w` | T2w | 0.639 | 0.696 |
| `data-multi-subject_sub-amu02_T2w` | T2w | 0.594 | 0.643 |
| `data-multi-subject_sub-barcelona01_T2w` | T2w | 0.677 | 0.724 |
| `data-multi-subject_sub-brnoUhb03_T2w` | T2w | 0.655 | 0.719 |
| `hc-leipzig-7t-mp2rage_sub-sspr17_inv-1_part-mag_MP2RAGE` | INV1 | 0.647 | 0.642 |
| `hc-leipzig-7t-mp2rage_sub-sspr17_inv-2_part-mag_MP2RAGE` | INV2 | 0.690 | 0.703 |
| `hc-leipzig-7t-mp2rage_sub-sspr17_UNIT1` | UNIT1 | 0.691 | 0.687 |
| `hc-leipzig-7t-mp2rage_sub-sspr24_inv-1_part-mag_MP2RAGE` | INV1 | 0.554 | 0.561 |
| `hc-leipzig-7t-mp2rage_sub-sspr24_inv-2_part-mag_MP2RAGE` | INV2 | 0.642 | 0.651 |
| `hc-leipzig-7t-mp2rage_sub-sspr24_UNIT1` | UNIT1 | 0.612 | 0.622 |
| `hc-leipzig-7t-mp2rage_sub-sspr31_inv-1_part-mag_MP2RAGE` | INV1 | 0.627 | 0.643 |
| `hc-leipzig-7t-mp2rage_sub-sspr31_inv-2_part-mag_MP2RAGE` | INV2 | 0.695 | 0.704 |
| `hc-leipzig-7t-mp2rage_sub-sspr31_UNIT1` | UNIT1 | 0.680 | 0.695 |
| `hc-leipzig-7t-mp2rage_sub-sspr37_inv-1_part-mag_MP2RAGE` | INV1 | 0.666 | 0.671 |
| `hc-leipzig-7t-mp2rage_sub-sspr37_inv-2_part-mag_MP2RAGE` | INV2 | 0.711 | 0.717 |
| `hc-leipzig-7t-mp2rage_sub-sspr37_UNIT1` | UNIT1 | 0.712 | 0.718 |

## Observations and limitations

- The best mean semantic Dice is on INV2 (`0.685`), followed by UNIT1 (`0.674`); INV1 is lower (`0.623`).
- T2w performance is more variable, with the lowest-scoring test image at `0.491` macro Dice.
- The model predicts C2–C8 in every image but never predicts T1. T1 is present in three T2w references, so all three T1 Dice values are 0. This needs investigation (in particular, class coverage and representation in the training folds).
- These are **fold-0-only** results. They should be considered preliminary until folds 1–4 are trained and ensembled/evaluated.
- The uncropped `MULTICON_v2` model was previously evaluated for accuracy on the same 17-image split in #84, but no usable cervical inference-time benchmark was found in the existing issues. The matched run above provides the first segmentation-only baseline and cropped comparison; repeated runs and the cropped end-to-end timing remain outstanding.
- The official `nnUNetv2_evaluate_folder` summary is retained, but its aggregate foreground Dice is `NaN` because absent per-image classes are not aggregated safely for this dataset. The tables above use an explicit NaN-safe calculation and report the metric definition.
- The timing above is not an end-to-end pipeline benchmark and cannot yet support a speedup claim. Detector time and uncropping/native-space export must be measured separately and included in the total.

## Visuals

### Training curves, fold 0

## Next steps

- [ ] Investigate the missing T1 predictions and inspect the three affected cases qualitatively.
- [x] Rerun the current uncropped model on the 17 full source images, record its exact checkpoint/folds, and evaluate it with the same NaN-safe script, including T1/label 9.
- [x] Complete the first matched segmentation-only timing run for the uncropped and cropped models.
- [ ] Repeat each segmentation-only arm at least twice more, alternating run order, and report median [IQR].
- [ ] Benchmark the cropped model end to end from the full source image, including detector/crop and native-grid restoration/export.
- [ ] Restore cropped predictions to the native full-image grid and report paired per-image Dice and timing differences.
- [ ] Train/evaluate folds 1–4 and report ensemble results.
- [ ] Compare cropped and uncropped ensembles once the remaining cropped folds are available.
- [ ] Add representative qualitative overlays (best, median, worst, and a T1 case).
- [ ] Report segmentation-only and end-to-end speedups; do not infer speedup from voxel counts alone.

Start with MP2RAGE_T2w_fold_splits.csv and the documented nnUNetv2_predict commands for the cropped and uncropped arms. Measure the cropped detector, preprocessing, inference, restoration/export, and total times over repeated alternating runs, then report median [IQR], batch totals, speedup, and peak GPU memory if available. Done means the end-to-end cropped timing is no longer TODO and is directly comparable with the uncropped baseline.

Written by the indexing model from the issue text.