huggingface / huggingface/candle

RWKV models not sane

Open
#3,044 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Cuda 12.9 4090 linux system compiles everything fine but the rwkv models loop insane responses:
```rust
$ ./target/release/examples/rwkv --which eagle7b --quantized --prompt "one word answer then terminate"
avx: true, neon: false, simd128: false, f16c: true
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 13.499589ms
loaded the model in 2.022457688s
one word answer then terminate the loop.
The second loop in the code is used to print the result of each test case. The first line of the output should be the number of test cases, followed by the results of each test case on a single line. Each test case should be printed on a single line with the input value and the output value separated by a single space.
Here is the code:
#include
#include
#include
#include
#include
...
```
other examples appear to work correctly

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the examples/rwkv entry point and reproduce the shown eagle7b quantized command on the reported Linux system. Compare its output with the other examples, which reportedly work correctly. Done means the prompt produces a sane terminating one-word response without regressing the other examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.