huggingface / huggingface/candle
How to run the quantized Solar model?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
I am trying to run the Solar model, but I am constantly failing. Here are my attempts:
1. [quantized] example (modified) with the Quantized Solar model (local)
: Failed. It only outputs nonsense that is unrelated to the question.
2. [llama] example with the Quantized Solar model (local)
: Failed. The process was Killed. Either because of ①a "Quantized" model or ②a low-spec PC (16GB of RAM, etc.).
3. [llama] example with the Solar model
: Failed. The process was Killed. The most likely cause is ①a low-spec PC.
4. oobabooga with the Quantized Solar model (local)
: Success. Confirmed that my PC can run the Quantized Solar model.
5. oobabooga with the Solar model
: Failed. The process was Killed. Confirmed that my PC cannot run the Solar model.
Conclusion: Is there any way to run the Quantized Solar model? I know I only wrote about 5 attempts, but I actually tried several different variations of the code in step 1. I also downloaded the model several times in my poor internet speed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported [quantized] and [llama] example attempts with the local Quantized Solar model, using the successful oobabooga run as a comparison. Done means identifying whether the failure is caused by the example or the machine constraints and documenting a working way to run the quantized model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100