microsoft / microsoft/BitNet

Can't convert hugging face model to tl2 GGUF

Open
#376 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

long story short
want to use the bitnet model with quant type tl2
the default gguf model on hf is i2_s

converting the model to tl2 myself throws the following error (this after I changed the BitnetForCausalLM from n to capital N ofc)
python3 ./utils/generate-dummy-bitnet-model.py --outtype tl2 --outfile ./ggml-model-tl2.gguf path/to/model/bitnet-b1.58-2B-4T


Traceback (most recent call last):
  File "/home/shubh/BitNet/./utils/generate-dummy-bitnet-model.py", line 1048, in <module>
    main()
  File "/home/shubh/BitNet/./utils/generate-dummy-bitnet-model.py", line 979, in main
    model_instance.set_vocab()
  File "/home/shubh/BitNet/./utils/generate-dummy-bitnet-model.py", line 794, in set_vocab
    self._set_vocab_sentencepiece()
  File "/home/shubh/BitNet/./utils/generate-dummy-bitnet-model.py", line 441, in _set_vocab_sentencepiece
    raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: /home/shubh/BitNet/HFmodel/bitnet-b1.58-2B-4T/tokenizer.model

looking at the code the sentencepiece requires tokenizer.model not tokenizer.json

    def _set_vocab_sentencepiece(self):
        from sentencepiece import SentencePieceProcessor

        tokenizer_path = self.dir_model / 'tokenizer.model'

Want to know how can I make this work
Is the model directly available somewhere, or is there a better to to generate tl2?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in utils/generate-dummy-bitnet-model.py, especially set_vocab() and _set_vocab_sentencepiece(), and inspect the Hugging Face model directory for the tokenizer files it expects. Reproduce the command with the BitNet model and verify that conversion with --outtype tl2 completes and produces ggml-model-tl2.gguf.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.