huggingface / huggingface/candle

How to run the quantized Solar model?

Open
#1,624 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I am trying to run the Solar model, but I am constantly failing. Here are my attempts:

1. [quantized] example (modified) with the Quantized Solar model (local)
: Failed. It only outputs nonsense that is unrelated to the question.
2. [llama] example with the Quantized Solar model (local)
: Failed. The process was Killed. Either because of ①a "Quantized" model or ②a low-spec PC (16GB of RAM, etc.).
3. [llama] example with the Solar model
: Failed. The process was Killed. The most likely cause is ①a low-spec PC.
4. oobabooga with the Quantized Solar model (local)
: Success. Confirmed that my PC can run the Quantized Solar model.
5. oobabooga with the Solar model
: Failed. The process was Killed. Confirmed that my PC cannot run the Solar model.

Conclusion: Is there any way to run the Quantized Solar model? I know I only wrote about 5 attempts, but I actually tried several different variations of the code in step 1. I also downloaded the model several times in my poor internet speed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported [quantized] and [llama] example attempts with the local Quantized Solar model, using the successful oobabooga run as a comparison. Done means identifying whether the failure is caused by the example or the machine constraints and documenting a working way to run the quantized model.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.