OpenBMB / OpenBMB/CPM-Bee

CPM Bee 微调时设置 half 出现 CUDA 报错,不设置 half 则 assert 报错

Open
#79 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.4k
Forks
176
PR merge metrics
No merged PRs in 30d

Description

CPM 使用微调脚本训练, 不开启 --use-delta 这一选项,则出现如下错误:
Traceback (most recent call last):
File "finetune_cpm_bee.py", line 503, in
main()
File "finetune_cpm_bee.py", line 499, in main
finetune(args, tokenizer, model, optimizer, lr_scheduler, optim_manager)
File "finetune_cpm_bee.py", line 364, in finetune
optim_manager.step()
File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/optim/optim_manager.py", line 131, in step
optimizer.step(scale=self.loss_scale)
File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/torch/optim/optimizer.py", line 109, in wrapper
return func(*args, **kwargs)
File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/optim/adam_offload.py", line 77, in step
state["_grad_fp16"] = torch.empty(p.size(), dtype=torch.float16, pin_memory=True) # on host
RuntimeError: CUDA error: OS call failed or operation not supported on this OS
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with finetune_cpm_bee.py, especially main and finetune around the reported optimizer step, and reproduce the run with and without half and --use-delta. Trace whether the failures come from the finetuning path or the bmtrain optimizer, and consider the issue complete when both reported configurations run without the CUDA or assertion errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.