[candle-onnx] simple_eval is hardcoded to CPU, preventing CUDA/GPU acceleration

The ``candle_onnx::simple_eval`` function is the primary public API for model execution, but it is hardcoded to use ``Device::Cpu``. It provides no way to specify a target device (CUDA/Metal).

Internally, ``get_tensor``, ``AttrOwned`` implementations, and various operators (like ``RandomNormal``, ``Gemm``, ``Constant``) explicitly use ``&Device::Cpu``. This forces all model weights and intermediate constants onto the CPU. When a user provides inputs on a GPU, the execution fails because ``candle`` cannot perform operations between tensors located on different devices (CPU vs CUDA).
### System information

- OS Platform and Distribution : Arch Linux
- ONNX version:
```
candle-core = { version = "0.10.2", features = ["cuda" ] }
candle-nn = { version = "0.10.2", features = ["cuda"] }
candle-onnx = "0.10.2"
```

### Reproduction instructions
1. Load an ONNX model using candle_onnx::read_file. (i use ``det_10g.onnx``)

https://github.com/deepinsight/insightface/releases
```
find ~/.insightface/models/buffalo_l/
.../.insightface/models/buffalo_l/
.../.insightface/models/buffalo_l/genderage.onnx
.../.insightface/models/buffalo_l/2d106det.onnx
.../.insightface/models/buffalo_l/det_10g.onnx
.../.insightface/models/buffalo_l/1k3d68.onnx
.../.insightface/models/buffalo_l/w600k_r50.onnx
```
2. Prepare input tensors on a CUDA device: ``Tensor::randn(..., &Device::new_cuda(0)?)``.
3. Attempt to run ``candle_onnx::simple_eval(&model, inputs)``.
Result: The process returns error indicating that tensors are on different devices , preventing GPU inference.

### Expected behavior
``simple_eval`` should accept a ``&Device`` parameter and propagate it throughout the evaluation process (to initializers, constants, and random generators) to allow full GPU acceleration.
### Notes
I have implemented a local Proof of Concept (PoC) by patching the eval module to propagate the Device parameter.
Performance impact on an InsightFace model:
- Before (hardcoded CPU): ~30 seconds per inference.
- After (patched for CUDA): ~2 seconds per inference.

I am attaching a ``git diff`` of my changes below.
Disclaimer: This diff is for demonstration purposes only. It was extracted from a modified local environment and is intended solely to show the logic required for device propagation.

```diff
diff --git a/src/candlefix/mod.rs b/src/candlefix/mod.rs
index 6d3027e..687cee8 100644
--- a/src/candlefix/mod.rs
+++ b/src/candlefix/mod.rs
@@ -35,5 +35,5 @@ trait Attr {
trait AttrOwned: Sized {
const TYPE: AttributeType;
- fn get(attr: &onnx::AttributeProto) -> Result;
+ fn get(attr: &onnx::AttributeProto, device: &Device) -> Result;
}

@@ -77,5 +77,5 @@ impl Attr for GraphProto {
impl AttrOwned for Vec {
const TYPE: AttributeType = AttributeType::Strings;
- fn get(attr: &onnx::AttributeProto) -> Result {
+ fn get(attr: &onnx::AttributeProto, device: &Device) -> Result {
let mut ret = vec![];
for bytes in attr.strings.iter() {
@@ -89,5 +89,5 @@ impl AttrOwned for Vec {
impl AttrOwned for Tensor {
const TYPE: AttributeType = AttributeType::Tensor;
- fn get(attr: &onnx::AttributeProto) -> Result {
+ fn get(attr: &onnx::AttributeProto, device: &Device) -> Result {
let tensor_proto = match &attr.t {
Some(value) => value,
@@ -127,5 +127,5 @@ impl AttrOwned for Tensor {
}

- Tensor::from_raw_buffer(&tensor_proto.raw_data, dtype, &dims, &Device::Cpu)
+ Tensor::from_raw_buffer(&tensor_proto.raw_data, dtype, &dims, device)
}
}
@@ -178,5 +178,9 @@ fn get_attr_opt<'a, T: Attr + ?Sized>(
}

-fn get_attr_opt_owned(node: &onnx::NodeProto, name: &str) -> Result> {
+fn get_attr_opt_owned(
+ node: &onnx::NodeProto,
+ name: &str,
+ device: &Device,
+) -> Result> {
match node.attribute.iter().find(|attr| attr.name == name) {
None => Ok(None),
@@ -190,5 +194,5 @@ fn get_attr_opt_owned(node: &onnx::NodeProto, name: &str) -> Resul
)
}
- let val = T::get(attr)?;
+ let val = T::get(attr, device)?;
Ok(Some(val))
}
@@ -196,5 +200,5 @@ fn get_attr_opt_owned(node: &onnx::NodeProto, name: &str) -> Resul
}

-pub fn get_tensor(t: &onnx::TensorProto, name: &str) -> Result {
+pub fn get_tensor(t: &onnx::TensorProto, name: &str, device: &Device) -> Result {
let dims: Vec = t.dims.iter().map(|&x| x as usize).collect();
match DataType::try_from(t.data_type) {
@@ -205,8 +209,8 @@ pub fn get_tensor(t: &onnx::TensorProto, name: &str) -> Result {
unsafe { std::slice::from_raw_parts(t.raw_data.as_ptr() as *const i32, len) };
let data = data.iter().map(|v| *v as i64).collect::>();
- Tensor::from_vec(data, len, &Device::Cpu)
+ Tensor::from_vec(data, len, device)
} else {
let data = t.int32_data.iter().map(|v| *v as i64).collect::>();
- Tensor::from_vec(data, t.int32_data.len(), &Device::Cpu)
+ Tensor::from_vec(data, t.int32_data.len(), device)
}
}
@@ -214,16 +218,11 @@ pub fn get_tensor(t: &onnx::TensorProto, name: &str) -> Result {
Some(dt) => {
if dt == DType::F32 && !t.float_data.is_empty() {
- Tensor::from_slice(&t.float_data, dims.as_slice(), &Device::Cpu)
+ Tensor::from_slice(&t.float_data, dims.as_slice(), device)
} else if dt == DType::F64 && !t.double_data.is_empty() {
- Tensor::from_slice(&t.double_data, dims.as_slice(), &Device::Cpu)
+ Tensor::from_slice(&t.double_data, dims.as_slice(), device)
} else if dt == DType::I64 && !t.int64_data.is_empty() {
- Tensor::from_slice(&t.int64_data, dims.as_slice(), &Device::Cpu)
+ Tensor::from_slice(&t.int64_data, dims.as_slice(), device)
} else {
- Tensor::from_raw_buffer(
- t.raw_data.as_slice(),
- dt,
- dims.as_slice(),
- &Device::Cpu,
- )
+ Tensor::from_raw_buffer(t.raw_data.as_slice(), dt, dims.as_slice(), device)
}
}
@@ -246,4 +245,5 @@ pub fn simple_eval(
model: &onnx::ModelProto,
mut inputs: HashMap,
+ device: &Device,
) -> Result> {
let graph = match &model.graph {
@@ -251,5 +251,5 @@ pub fn simple_eval(
Some(graph) => graph,
};
- simple_eval_(graph, &mut inputs)
+ simple_eval_(graph, &mut inputs, device)
}

@@ -257,7 +257,8 @@ fn simple_eval_(
graph: &onnx::GraphProto,
values: &mut HashMap,
+ device: &Device,
) -> Result> {
for t in graph.initializer.iter() {
- let tensor = get_tensor(t, t.name.as_str())?;
+ let tensor = get_tensor(t, t.name.as_str(), device)?;
values.insert(t.name.to_string(), tensor);
}
@@ -584,9 +585,6 @@ fn simple_eval_(
"ConstantOfShape" => {
let input = get(&node.input[0])?;
- let value = get_attr_opt_owned::(node, "value")?.unwrap_or(Tensor::zeros(
- (),
- DType::F32,
- &Device::Cpu,
- )?);
+ let value = get_attr_opt_owned::(node, "value", device)?
+ .unwrap_or(Tensor::zeros((), DType::F32, device)?);

let shape_vec: Vec = input
@@ -771,5 +769,5 @@ fn simple_eval_(
to_vec0_flexible::<$t>(limit)?,
to_vec0_flexible::<$t>(delta)?,
- &Device::Cpu,
+ device,
)?
};
@@ -1096,5 +1094,5 @@ fn simple_eval_(
AttributeType::Tensor => {
let t = value.t.as_ref().unwrap();
- get_tensor(t, &node.name)?
+ get_tensor(t, &node.name, device)?
}
rtype => bail!("unsupported 'value' type {rtype:?} for {}", node.name),
@@ -1172,5 +1170,5 @@ fn simple_eval_(
);
}
- let branch_out = simple_eval_(sub_graph, values)?;
+ let branch_out = simple_eval_(sub_graph, values, device)?;
for (i, out) in node.output.iter().enumerate() {
values.insert(
@@ -1710,9 +1708,9 @@ fn simple_eval_(
let low: f32 = get_attr_opt(node, "low")?.copied().unwrap_or(0.0);
let high: f32 = get_attr_opt(node, "high")?.copied().unwrap_or(1.0);
- Tensor::rand(low, high, shape, &Device::Cpu)?.to_dtype(dtype)?
+ Tensor::rand(low, high, shape, device)?.to_dtype(dtype)?
} else {
let mean: f32 = get_attr_opt(node, "mean")?.copied().unwrap_or(0.0);
let scale: f32 = get_attr_opt(node, "scale")?.copied().unwrap_or(1.0);
- Tensor::randn(mean, scale, shape, &Device::Cpu)?.to_dtype(dtype)?
+ Tensor::randn(mean, scale, shape, device)?.to_dtype(dtype)?
};
values.insert(node.output[0].clone(), output);
@@ -1808,6 +1806,6 @@ fn simple_eval_(
let beta = get_attr_opt::(node, "beta")?.copied().unwrap_or(1.0);

- let alpha = Tensor::full(alpha, a.shape(), &Device::Cpu)?;
- let beta = Tensor::full(beta, c.shape(), &Device::Cpu)?;
+ let alpha = Tensor::full(alpha, a.shape(), device)?;
+ let beta = Tensor::full(beta, c.shape(), device)?;

let trans_a = get_attr_opt::(node, "transA")?.copied().unwrap_or(0);
@@ -1839,5 +1837,5 @@ fn simple_eval_(
"Tanh".to_string(),
];
- let activations = get_attr_opt_owned::>(node, "activations")?
+ let activations = get_attr_opt_owned::>(node, "activations", device)?
.unwrap_or(activations_default.clone());
if activations != activations_default {
@@ -2063,5 +2061,5 @@ fn simple_eval_(
// activation_alpha and activation_beta don't apply to (Tanh, Tanh) so ignoring them is okay
let activations_default = vec!["Tanh".to_string(), "Tanh".to_string()];
- let activations = get_attr_opt_owned::>(node, "activations")?
+ let activations = get_attr_opt_owned::>(node, "activations", device)?
.unwrap_or(activations_default.clone());
let clip = get_attr_opt::(node, "clip")?.copied();
```
---
# P.S. This issue report and the technical description were prepared with the assistance of an AI (LLM) to ensure clarity and proper formatting, based on my local findings and successful performance tests.

Start by reading src/candlefix/mod.rs and the simple_eval entry point, tracing get_tensor, attribute handling, initializers, constants, random operators, and nested graph evaluation. Run the reported det_10g.onnx reproduction with CUDA inputs. Done means model tensors and generated values consistently use the selected device and GPU inference succeeds without CPU/CUDA device errors.

Written by the indexing model from the issue text.