i2_s quantized model giving random outputs after fine-tuning.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
I have fine-tuned the bitnet_b1_58-large (https://huggingface.co/1bitLLM/bitnet_b1_58-large) on the Alpaca Instruction Tuning dataset. After conversion, the f32.gguf model is giving proper results. But the i2_s.gguf is just outputting random tokens. Hopefully, the conversion process is correct because the FP32 model is giving correct results. Do I need to manage something or am I missing something when converting custom fine-tuned models?
Following are some results that I am getting using the i2_s model:
### Instruction:
Write about the following topic.
### Input:
Deep Learning
### Response:
Deep Learning ath swe shortNC rev rest throwiseë co /**ab symbols symbolay groundë class strikingast '''rob conjug Search shadow rep lath shadow a'ewewunnwise shadow rep ground ground ground ground ground ground ground throwiserobosesrob whatever shadow by ground ground ground groundew style ground ground ground ground ground groundbody whom rang ground ground ground ground ground ground ground ground ground ground groundew rang groundewoi control rest ground groundew rangiz shadow houredaburgeda a ground ground ground ground ground ground ground ground ground ground ground foodilë shell contactellite reception’ew swearation pro work shadow icon' ritane rangage
It should have been similar to this (f32.gguf model).
### Instruction:
Write about the following topic.
### Input:
Deep Learning
### Response:
Deep Learning is a technique used by computers to learn complex patterns, data and patterns in large amounts of data. It involves using a combination of techniques such as machine learning and deep learning, which can help learn complex patterns and identify patterns in large datasets
Is there an issue with the tokenizer, or something else? Any help is appreciated.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported conversion from the fine-tuned bitnet_b1_58-large model and compare the f32.gguf and i2_s.gguf outputs. Investigate whether the tokenizer or quantized conversion accounts for the random tokens; done means identifying the cause and documenting or correcting the conversion path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100