microsoft / microsoft/BitNet

failed to run benchmark for bitnet_b1_58

Open
#22 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

I followed the instructions in https://github.com/microsoft/BitNet?tab=readme-ov-file#benchmark

I downloaded bitnet_b1_58-large, however running the benchmark with
python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M
failed:

(bitnet-cpp) PS C:\work\microsoft\BitNet> python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M
Traceback (most recent call last):
File "C:\work\microsoft\BitNet\utils\generate-dummy-bitnet-model.py", line 1048, in
main()
File "C:\work\microsoft\BitNet\utils\generate-dummy-bitnet-model.py", line 971, in main
model_class = Model.from_model_architecture(hparams["architectures"][0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\work\microsoft\BitNet\utils\generate-dummy-bitnet-model.py", line 312, in from_model_architecture
raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'LlamaModel' not supported!

is there a necessary dependency missing / not installed ?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the benchmark instructions in the README and inspect utils/generate-dummy-bitnet-model.py, especially Model.from_model_architecture and the main entry point shown in the traceback. Determine whether the downloaded model's LlamaModel architecture should be supported or whether a dependency is missing. Done means the supplied command runs without the NotImplementedError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.