microsoft / microsoft/BitNet

i2_s quantized model giving random outputs after fine-tuning.

Open
#107 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

I have fine-tuned the bitnet_b1_58-large (https://huggingface.co/1bitLLM/bitnet_b1_58-large) on the Alpaca Instruction Tuning dataset. After conversion, the f32.gguf model is giving proper results. But the i2_s.gguf is just outputting random tokens. Hopefully, the conversion process is correct because the FP32 model is giving correct results. Do I need to manage something or am I missing something when converting custom fine-tuned models?

Following are some results that I am getting using the i2_s model:

 ### Instruction:
Write about the following topic.

### Input:
Deep Learning

### Response:
Deep Learning ath swe shortNC rev rest throwiseë co /**ab symbols symbolay groundë class strikingast '''rob conjug Search shadow rep lath shadow a'ewewunnwise shadow rep ground ground ground ground ground ground ground throwiserobosesrob whatever shadow by ground ground ground groundew style ground ground ground ground ground groundbody whom rang ground ground ground ground ground ground ground ground ground ground groundew rang groundewoi control rest ground groundew rangiz shadow houredaburgeda a ground ground ground ground ground ground ground ground ground ground ground foodilë shell contactellite reception’ew swearation pro work shadow icon' ritane rangage

It should have been similar to this (f32.gguf model).

 ### Instruction:
Write about the following topic.

### Input:
Deep Learning

### Response:
Deep Learning  is a technique used by computers to learn complex patterns, data and patterns in large amounts of data. It involves using a combination of techniques such as machine learning and deep learning, which can help learn complex patterns and identify patterns in large datasets

Is there an issue with the tokenizer, or something else? Any help is appreciated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported conversion from the fine-tuned bitnet_b1_58-large model and compare the f32.gguf and i2_s.gguf outputs. Investigate whether the tokenizer or quantized conversion accounts for the random tokens; done means identifying the cause and documenting or correcting the conversion path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.