Question: Why does USE_PYTORCH_FOR_PREPROCESSING default to False?

Question: Why does USE_PYTORCH_FOR_PREPROCESSING default to False?

Description

USE_PYTORCH_FOR_PREPROCESSING (env.py:329-331) defaults to False, meaning all image preprocessing (resize, letterbox, BGR→RGB, normalize) runs on CPU via OpenCV even when a CUDA GPU is available and selected as the ONNX Runtime execution provider.

The GPU preprocessing path exists and is fully wired — preprocess.py, roboflow.py, object_detection_base.py, classification_base.py, rfdetr.py, and yolact_instance_segmentation.py all have if USE_PYTORCH_FOR_PREPROCESSING: branches that use torch tensors for resize/normalize. But it's opt-in rather than opt-out.

I'd like to understand the reasoning behind the default before proposing a change. Is the CPU default driven by:

• Stability concerns with the torch preprocessing path on certain hardware?
• Memory overhead from keeping tensors on GPU?
• Compatibility with CPU-only deployments where torch import would be wasted?
• Historical reasons that may no longer apply?

Use case

While profiling inference throughput across 4 different NVIDIA GPUs (RTX PRO 4500 Blackwell, RTX 2080 Ti 22GB, RTX 2080 Ti 11GB, RTX 4060 Ti), I measured the preprocessing stage in isolation:

Each result is the mean of 100 iterations after 5 warmup iterations, repeated across 3 runs per GPU (12 data points total). The speedup reproduces on every GPU tested, with a minimum of 11x.

For a 30 FPS camera stream, CPU preprocessing at ~1.5 ms/frame consumes ~4.5% of the frame budget. For a 10-image batch request, it's ~15 ms — a meaningful fraction of total request latency when the model inference itself is 2-10 ms on GPU.

If there's no blocking reason for the CPU default, enabling GPU preprocessing by default when CUDA is available would be a significant latency win. If there IS a blocking reason, I'd like to understand it so the issue can be scoped correctly — e.g., maybe the default should change only when CUDAExecutionProvider is active, or maybe a runtime auto-detection path is better than a static default.

Additional

• The flag was introduced with the tensor-native preprocessing work and the RF-DETR GPU preprocessing path. The rfdetr.py module has the most extensive GPU preprocessing coverage (9 conditional branches).
• ENABLE_TENSOR_DATA_REPRESENTATION (env.py:1440-1460) is a related flag that keeps image data as tensors end-to-end. It also defaults to False.
• The profiling was done with a synthetic 4-layer ConvNet ONNX model (640×640 input) to isolate preprocessing from model-specific behavior. Full methodology and raw JSON results are available.

Are you willing to submit a PR?

Yes I'd like to help by submitting a PR!

Start with env.py:329-331 and trace the USE_PYTORCH_FOR_PREPROCESSING branches in preprocess.py, roboflow.py, object_detection_base.py, classification_base.py, rfdetr.py, and yolact_instance_segmentation.py. Compare the related ENABLE_TENSOR_DATA_REPRESENTATION setting in env.py:1440-1460 and existing deployment behavior. Done means establishing the rationale for the default and agreeing on a scoped change or documenting why none should be made.