huggingface / huggingface/candle

some token duplicated in candle-examples trocr

Open
#2,302 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hello, there appears to be a bug in the trocr example. Possibly with the way images are tokenized.

For example, this image produces 754754.7 instead of 754.7

bug

I have added some debug print statements and I notice that the token 39382 associated with 754 appears twice (at start_pos 1 and start_pos 2).

```
cargo run --example trocr --release -- --which base-printed --image candle-examples/examples/trocr/assets/bug.png
Finished release [optimized] target(s) in 3.11s
Running `target/release/examples/trocr --which base-printed --image candle-examples/examples/trocr/assets/bug.png`
Running on CPU, to run on GPU, build this example with `--features cuda`
model: "/home/xxx/.cache/huggingface/hub/models--microsoft--trocr-base-printed/snapshots/24216f24cd78fe1a9c8b4e6e4565aec5c9220e63/model.safetensors"
context_size - 1 -
start_pos - 0 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 39382 -
t: 754
------------
context_size - 1 -
start_pos - 1 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 39382 -
t: 754
------------
context_size - 1 -
start_pos - 2 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 4 -
------------
context_size - 1 -
start_pos - 3 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 406 -
t: .7
------------
context_size - 1 -
start_pos - 4 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 821 -
t: g
------------
context_size - 1 -
start_pos - 5 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 2 -
------------
```

On the other hand, an almost identical image that begins and ends with a 2 instead of a 7 is OK.

nobug

```
cargo run --example trocr --release -- --which base-printed --image candle-examples/examples/trocr/assets/nobug.png
Finished release [optimized] target(s) in 3.32s
Running `target/release/examples/trocr --which base-printed --image candle-examples/examples/trocr/assets/nobug.png`
Running on CPU, to run on GPU, build this example with `--features cuda`
model: "/home/xxx/.cache/huggingface/hub/models--microsoft--trocr-base-printed/snapshots/24216f24cd78fe1a9c8b4e6e4565aec5c9220e63/model.safetensors"
context_size - 1 -
start_pos - 0 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 30959 -
t: 254
------------
context_size - 1 -
start_pos - 1 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 4 -
------------
context_size - 1 -
start_pos - 2 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 176 -
t: .2
------------
context_size - 1 -
start_pos - 3 -
input_ids: - Tensor[dims 1, 1; u32] -
token - 2 -
------------
```

The model itself appears to be fine. Checked out like this using the API:

Screenshot 2024-06-30 133714

Just in case there is some uncertainty about the "g" being a "g" or an "8" and maybe that's throwing it off, I tried it without the "g". Here is what we have (no debug print): again 754 segment is there twice.

nog

```
cargo run --example trocr --release -- --which base-printed --image candle-examples/examples/trocr/assets/nog.png
Compiling candle-examples v0.6.0 (/mnt/c/Users/xxx/RUST/candle/candle-examples)
Finished release [optimized] target(s) in 26.00s
Running `target/release/examples/trocr --which base-printed --image candle-examples/examples/trocr/assets/nog.png`
Running on CPU, to run on GPU, build this example with `--features cuda`
model: "/home/xxx/.cache/huggingface/hub/models--microsoft--trocr-base-printed/snapshots/24216f24cd78fe1a9c8b4e6e4565aec5c9220e63/model.safetensors"
encoder_xs: Tensor[dims 1, 577, 768; f32]
754754.7
```

I'm not sure if there is an extra token being passed to the model, or the returning token from the model being accessed twice. Maybe it has something to do with the differences between the Rust image library and whatever they use for image parsing on huggingface.co API.
But what makes the digit 7 special anyway? And any hint where to look next?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the candle-examples/examples/trocr entry point and reproduce the issue using the reported cargo run command with bug.png, then compare its debug output with nobug.png and nog.png. Trace how input_ids, start_pos, and returned tokens are handled around the duplicated 754 token, including the example's image processing path. Done means the affected images no longer produce duplicated text while the existing comparison image still decodes correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.