EleutherAI / EleutherAI/math-lm

cannot convert raw llama weights to NeoX

Open
#96 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
95
PR merge metrics
No merged PRs in 30d

Description

Hello! Thanks for your great work, but I met some problems when trying to replicate the results.

Specifically, I cannot find convert_raw_llama_weights_to_hf.py as depicted in [README.md](https://github.com/EleutherAI/gpt-neox/tree/5dd366539803dbf1fd725cc057013fd002a4cfd4#converting-llama-2-checkpoints-into-neox-format) .

However, I found [convert_raw_llama_weights_to_neox.py](https://github.com/EleutherAI/gpt-neox/blob/5dd366539803dbf1fd725cc057013fd002a4cfd4/tools/convert_raw_llama_weights_to_neox.py), which seems can convert Meta->NeoX format.

But the python script doesn't support `--config_file`, so I use `--model_size=7B` instead. Unfortunately, I met an error:

```sh
(gptneox2) root@ebf9662e:/math-lm# bash convert.sh
sequential
dict_keys(['dim', 'n_layers', 'n_heads', 'multiple_of', 'ffn_dim_multiplier', 'norm_eps', 'rope_theta'])
Traceback (most recent call last):
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 650, in
main()
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 641, in main
convert_model_sequential(
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 308, in convert_model_sequential
num_kv_heads = params["n_kv_heads"]
KeyError: 'n_kv_heads'
```

Here is my convert.sh:
```sh
python convert_raw_codellama_weights_to_neox.py \
--input_dir /math-lm/codellama \
--model_size 7B \
--output_dir /math-lm/codellama/7B-NeoX \
--num_output_shards 2
```

Here is my raw codellama7b:
```sh
(gptneox2) root@ebf9662e:/math-lm/codellama/7B# tree .
.
├── checklist.chk
├── consolidated.00.pth
├── params.json
└── tokenizer.model
```

Looking forward to your reply, any help will be appreciated!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.