EleutherAI / EleutherAI/math-lm
cannot convert raw llama weights to NeoX
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 95
- PR merge metrics
- No merged PRs in 30d
Description
Hello! Thanks for your great work, but I met some problems when trying to replicate the results.
Specifically, I cannot find convert_raw_llama_weights_to_hf.py as depicted in [README.md](https://github.com/EleutherAI/gpt-neox/tree/5dd366539803dbf1fd725cc057013fd002a4cfd4#converting-llama-2-checkpoints-into-neox-format) .
However, I found [convert_raw_llama_weights_to_neox.py](https://github.com/EleutherAI/gpt-neox/blob/5dd366539803dbf1fd725cc057013fd002a4cfd4/tools/convert_raw_llama_weights_to_neox.py), which seems can convert Meta->NeoX format.
But the python script doesn't support `--config_file`, so I use `--model_size=7B` instead. Unfortunately, I met an error:
```sh
(gptneox2) root@ebf9662e:/math-lm# bash convert.sh
sequential
dict_keys(['dim', 'n_layers', 'n_heads', 'multiple_of', 'ffn_dim_multiplier', 'norm_eps', 'rope_theta'])
Traceback (most recent call last):
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 650, in
main()
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 641, in main
convert_model_sequential(
File "gpt-neox/tools/convert_raw_codellama_weights_to_neox.py", line 308, in convert_model_sequential
num_kv_heads = params["n_kv_heads"]
KeyError: 'n_kv_heads'
```
Here is my convert.sh:
```sh
python convert_raw_codellama_weights_to_neox.py \
--input_dir /math-lm/codellama \
--model_size 7B \
--output_dir /math-lm/codellama/7B-NeoX \
--num_output_shards 2
```
Here is my raw codellama7b:
```sh
(gptneox2) root@ebf9662e:/math-lm/codellama/7B# tree .
.
├── checklist.chk
├── consolidated.00.pth
├── params.json
└── tokenizer.model
```
Looking forward to your reply, any help will be appreciated!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.