LAION-AI / LAION-AI/Open-Assistant
Supervised fine-tuning: "RuntimeError: expected scalar type Half but found Float" during evaluation
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
While running supervised fine-tuning with
python trainer_sft.py --configs lora-llama-13b webgpt_dataset_only
and the following config
lora-llama-13b:
dtype: fp16
log_dir: "llama_lora_log_13b"
learning_rate: 5e-5
model_name: openlm-research/open_llama_13b
output_dir: llama_model_13b_lora
weight_decay: 0.0
max_length: 2048
warmup_steps: 300
gradient_checkpointing: true
gradient_accumulation_steps: 1
per_device_train_batch_size: 6
per_device_eval_batch_size: 1
eval_steps: 500
num_train_epochs: 12
save_total_limit: 2
save_strategy: epoch
use_flash_attention: True
residual_dropout: 0.0
deepspeed_config: configs/zero_config.json
peft_model: true
peft_type: "lora"
use_custom_sampler: true
training runs fine but evaluation raises the following error (at the first eval step):
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/trainer.py", line 2234, in _maybe_log_save_evaluate
metrics = self.evaluate(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/trainer.py", line 2939, in evaluate
output = eval_loop(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/trainer.py", line 3120, in evaluation_loop
loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
File "/home/tgervet/Open-Assistant/model/model_training/trainer_sft.py", line 107, in prediction_step
loss, logits, labels, labels_mask = self._compute_loss(model, inputs)
File "/home/tgervet/Open-Assistant/model/model_training/trainer_sft.py", line 87, in _compute_loss
outputs = model(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/peft/peft_model.py", line 530, in forward
return self.base_model(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 687, in forward
outputs = self.model(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 577, in forward
layer_outputs = decoder_layer(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/Open-Assistant/model/model_training/models/patching_llama.py", line 28, in llama_forward_with_flash_attn
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/peft/tuners/lora.py", line 350, in forward
result += self.lora_B(self.lora_A(self.lora_dropout(x))) * self.scaling
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/tgervet/miniconda3/envs/open-assistant/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: expected scalar type Half but found Float
with environment
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.21.0 pypi_0 pypi
aiohttp 3.8.4 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
beautifulsoup4 4.12.2 pypi_0 pypi
bitsandbytes 0.40.0.post4 pypi_0 pypi
brotli 1.0.9 pypi_0 pypi
bzip2 1.0.8 h7b6447c_0
ca-certificates 2023.5.7 hbcca054_0 conda-forge
cattrs 23.1.2 pypi_0 pypi
certifi 2023.5.7 pypi_0 pypi
charset-normalizer 3.2.0 pypi_0 pypi
click 8.1.5 pypi_0 pypi
cmake 3.26.4 pypi_0 pypi
cudatoolkit-dev 11.7.0 h1de0b5d_6 conda-forge
datasets 2.13.1 pypi_0 pypi
deepspeed 0.9.5 pypi_0 pypi
dill 0.3.6 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
einops 0.6.1 pypi_0 pypi
evaluate 0.4.0 pypi_0 pypi
exceptiongroup 1.1.2 pypi_0 pypi
fastlangid 1.0.11 pypi_0 pypi
fasttext 0.9.2 pypi_0 pypi
filelock 3.12.2 pypi_0 pypi
flash-attn 1.0.8 pypi_0 pypi
frozenlist 1.4.0 pypi_0 pypi
fsspec 2023.6.0 pypi_0 pypi
gdown 4.7.1 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.32 pypi_0 pypi
grpcio 1.51.3 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
huggingface-hub 0.16.4 pypi_0 pypi
idna 3.4 pypi_0 pypi
inflate64 0.3.1 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.3.1 pypi_0 pypi
jsonschema 4.18.3 pypi_0 pypi
jsonschema-specifications 2023.6.1 pypi_0 pypi
langcodes 3.3.0 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
lit 16.0.6 pypi_0 pypi
loguru 0.6.0 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
model-training 1.0.0 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
msgpack 1.0.5 pypi_0 pypi
multidict 6.0.4 pypi_0 pypi
multiprocess 0.70.14 pypi_0 pypi
multivolumefile 0.2.3 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.1 pypi_0 pypi
ninja 1.11.1 pypi_0 pypi
nltk 3.8.1 pypi_0 pypi
numpy 1.25.1 pypi_0 pypi
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi
oasst-data 1.0.0 pypi_0 pypi
openssl 3.0.9 h7f8727e_0
packaging 23.1 pypi_0 pypi
pandas 2.0.3 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
peft 0.2.0 pypi_0 pypi
pip 23.1.2 py310h06a4308_0
protobuf 4.23.4 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
py7zr 0.20.5 pypi_0 pypi
pyarrow 12.0.1 pypi_0 pypi
pybcj 1.0.1 pypi_0 pypi
pybind11 2.10.4 pypi_0 pypi
pycryptodomex 3.18.0 pypi_0 pypi
pydantic 1.10.7 pypi_0 pypi
pygments 2.15.1 pypi_0 pypi
pyppmd 1.0.0 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
python 3.10.12 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi
python-rapidjson 1.10 pypi_0 pypi
pytz 2023.3 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzstd 0.15.9 pypi_0 pypi
ray 2.5.1 pypi_0 pypi
readline 8.2 h5eee18b_0
referencing 0.29.1 pypi_0 pypi
regex 2023.6.3 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
responses 0.18.0 pypi_0 pypi
rich 13.4.2 pypi_0 pypi
rpds-py 0.8.10 pypi_0 pypi
scikit-learn 1.3.0 pypi_0 pypi
scipy 1.11.1 pypi_0 pypi
sentencepiece 0.1.99 pypi_0 pypi
sentry-sdk 1.28.1 pypi_0 pypi
setproctitle 1.3.2 pypi_0 pypi
setuptools 67.8.0 py310h06a4308_0
six 1.16.0 pypi_0 pypi
smmap 5.0.0 pypi_0 pypi
soupsieve 2.4.1 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
sympy 1.12 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
texttable 1.6.7 pypi_0 pypi
threadpoolctl 3.2.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tokenizers 0.13.3 pypi_0 pypi
torch 2.0.1 pypi_0 pypi
torchtyping 0.1.4 pypi_0 pypi
tqdm 4.65.0 pypi_0 pypi
transformers 4.28.0.dev0 pypi_0 pypi
triton 2.0.0 pypi_0 pypi
tritonclient 2.35.0 pypi_0 pypi
trlx 0.7.0 pypi_0 pypi
typeguard 4.0.0 pypi_0 pypi
typing-extensions 4.7.1 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 2.0.3 pypi_0 pypi
wandb 0.15.5 pypi_0 pypi
wheel 0.38.4 py310h06a4308_0
xxhash 3.2.0 pypi_0 pypi
xz 5.4.2 h5eee18b_0
yarl 1.9.2 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
Any idea what could be causing this and how to fix it?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with model/model_training/trainer_sft.py, especially prediction_step and _compute_loss, then inspect model/model_training/models/patching_llama.py where the evaluation traceback reaches the LoRA projection. Reproduce the command with the lora-llama-13b and webgpt_dataset_only configurations, and verify that the first evaluation step completes without the Half/Float RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100