huggingface / huggingface/candle

Error when eval a quantized ONNX model

Open
#1,673 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

model file: `https://huggingface.co/Xenova/depth-anything-small-hf/blob/main/onnx/model_quantized.onnx`
main.rs:
```Rust
use std::collections::HashMap;

use candle_onnx::{self as onnx};

const MODEL: &str = "../depth-anything-small-hf/onnx/model_quantized.onnx";
// const MODEL: &str = "../depth-anything-small-hf/onnx/model.onnx";

fn main() {
#[cfg(debug_assertions)]
std::env::set_var("RUST_BACKTRACE", "1");

match _main() {
Ok(_) => (),
Err(err) => {
eprint!("{:?}", err);
}
}
}

fn _main() -> anyhow::Result<()> {
let model = onnx::read_file(MODEL)?;
let mut inputs = HashMap::new();
let output = match onnx::simple_eval(&model, inputs) {
Ok(ret) => ret,
Err(err) => {
if let candle_core::Error::WithBacktrace { inner, .. } = &err {
match &**inner {
candle_core::Error::ShapeMismatch { buffer_size, shape } => {
eprintln!("buffer_size: {}", buffer_size);
eprintln!("shape: {:?}", shape);
eprintln!("shape.elem_count: {}", shape.elem_count());
}
_ => (),
}
}
return Err(err.into());
}
};
Ok(())
}
```
result:
```
buffer_size: 0
shape: []
shape.elem_count: 1
Shape mismatch, got buffer of size 0 which is compatible with shape []
0: std::backtrace_rs::backtrace::libunwind::trace
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: std::backtrace::Backtrace::create
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/backtrace.rs:331:13
3: candle_core::error::Error::bt
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-core-0.3.3/src/error.rs:227:25
4: candle_core::tensor::Tensor::new_impl
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-core-0.3.3/src/tensor.rs:351:24
5: candle_core::tensor::Tensor::from_slice
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-core-0.3.3/src/tensor.rs:452:9
6: candle_core::safetensors::convert_slice
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-core-0.3.3/src/safetensors.rs:95:9
7: candle_core::safetensors::::from_raw_buffer
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-core-0.3.3/src/safetensors.rs:188:26
8: candle_onnx::eval::get_tensor
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-onnx-0.3.3/src/eval.rs:125:21
9: candle_onnx::eval::simple_eval
at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/candle-onnx-0.3.3/src/eval.rs:158:22
10: depth_estimation::_main
at ./src/main.rs:23:24
11: depth_estimation::main
at ./src/main.rs:12:11
12: core::ops::function::FnOnce::call_once
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/core/src/ops/function.rs:250:5
13: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/sys_common/backtrace.rs:155:18
14: std::rt::lang_start::{{closure}}
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/rt.rs:166:18
15: core::ops::function::impls:: for &F>::call_once
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/core/src/ops/function.rs:284:13
16: std::panicking::try::do_call
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panicking.rs:554:40
17: std::panicking::try
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panicking.rs:518:19
18: std::panic::catch_unwind
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panic.rs:142:14
19: std::rt::lang_start_internal::{{closure}}
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/rt.rs:148:48
20: std::panicking::try::do_call
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panicking.rs:554:40
21: std::panicking::try
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panicking.rs:518:19
22: std::panic::catch_unwind
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/panic.rs:142:14
23: std::rt::lang_start_internal
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/rt.rs:148:20
24: std::rt::lang_start
at /rustc/5d3d3479d774754856db2db3e439dfb88ef3c52f/library/std/src/rt.rs:165:17
25: _main
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure from main.rs using the linked quantized and unquantized model files, then inspect candle_onnx::eval::simple_eval and get_tensor, along with candle_core::safetensors::from_raw_buffer. Done means the quantized model evaluates without the reported zero-length buffer and shape mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.