Project-MONAI / Project-MONAI/monai-deploy-app-sdk

DICOM slope/intercept rescale can silently leave float64 data, causing a hard crash at inference (no forced float32 cast anywhere in the SDK)

Open
#592 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
138
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Summary

Follow-up on the data-type casting discussion from the Aug 6, 2025 WG call (Elan/Ming/Bryan/Will,
re: the PID-icon rescale fix and unsigned-int64/float64 concerns) — confirming with a concrete repro
what was flagged as a suspected memory-pressure risk. It's actually worse: a hard crash, not just an
OOM risk.

Root cause chain

  1. dicom_series_to_volume_operator.py, _get_rescaled_pixel_array (lines ~181-198): after applying
    rescale slope/intercept, it tries to downcast to uint8/uint16/int16/float32 in sequence.
    If the rescaled values don't cleanly match any of those exactly, it falls through and the array
    stays float64 — only logged at debug level ("Rescaled pixel data remains as of type float64"),
    no warning, no forced cast.
  2. MonaiSegInferenceOperator.pre_process (monai_seg_inference_operator.py) is abstract — there is no
    SDK-level guarantee that any application's pre-transforms cast the tensor to float32 before it
    reaches sliding_window_inference. Whether the crash below is hit depends entirely on whether the
    downstream app's own transforms happen to include a cast.
  3. When they don't, a float64 tensor reaches a float32-weighted model directly.

Repro of step 3 (confirms the failure mode)

import torch
from monai.inferers import sliding_window_inference
from monai.networks.nets import SegResNet

img_f64 = torch.rand((1, 1, 48, 48, 32), dtype=torch.float64)
model = SegResNet(spatial_dims=3, in_channels=1, out_channels=2).eval()

with torch.no_grad():
    sliding_window_inference(inputs=img_f64, roi_size=(96, 96, 96), sw_batch_size=1, predictor=model, overlap=0.25)

Result:

RuntimeError: expected scalar type Double but found Float

Not an OOM/memory-pressure issue as discussed — a deterministic crash whenever the upstream rescale
falls through to the float64 branch and the app's own pre-transforms don't happen to cast it back down.

Suggested handling

Open for discussion, a few options:

  1. Force-cast to float32 at the end of _get_rescaled_pixel_array when none of the exact-match
    branches apply, rather than silently leaving float64 (this is the simplest fix and matches the
    "float32 is standard for DL inference" guidance already discussed in the WG).
  2. If precision loss on fallback is a concern, at minimum raise the log level from debug to
    warning so this isn't silent, and document the expectation that downstream apps must cast
    explicitly.
  3. Consider whether MonaiSegInferenceOperator should validate/cast input dtype defensively before
    calling sliding_window_inference, independent of what an app's pre_process does.

Happy to submit a PR for whichever direction the team prefers — my inclination is option 1, it's the
smallest, safest fix and matches the team's own stated preference for float32 at inference time.

Environment: torch 2.8.0+cpu, monai 1.6.0, Python 3.13, Windows.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read _get_rescaled_pixel_array in dicom_series_to_volume_operator.py and MonaiSegInferenceOperator.pre_process in monai_seg_inference_operator.py, then run the supplied float64 sliding_window_inference repro. Confirm which handling option the maintainers want, and verify that rescaled data cannot silently cause the documented inference dtype crash under the agreed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.