OpenBMB / OpenBMB/CPM-Bee

微调报错CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Open
#73 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.4k
Forks
176
No merged PRs in 30d

Description

当设置--max-length 64时，运行不报错，但results目录下也没有结果，当设置--max-length 128时有报错。

单机1张3080 Ti卡，其中机器环境：

torch                   1.13.1+cu117
bmtrain                 0.2.1
NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7

微调指令：
torchrun --nnodes=1 --nproc_per_node=1 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:12345 finetune_cpm_bee.py --use-delta --model-config config/cpm-bee-2b.json --dataset ccpm_example/bin_data/train --eval_dataset ccpm_example/bin_data/eval --epoch 1 --batch-size 1 --train-iters 100 --save-name cpm_bee_finetune --max-length 128 --save results/ --lr 0.0001 --inspect-iters 100 --warmup-iters 1 --eval-interval 50 --early-stop-patience 5 --lr-decay-style noam --weight-decay 0.01 --clip-grad 1.0 --loss-scale 32768 --start-step 0 --load ckpts/pytorch_model.bin

微调过程中报错：

====================== Initialization ======================
rank :          0
local_rank :    0
world_size :    1
local_size :    1
master :        Goku:41629
device :        0
cpus :          [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

root
├── encoder (Encoder)
│   ├── layers (TransformerBlockList)
│   │   ├── 0-0,2-2,8-9,14-14,16-16,20-20,24-24,26-26,29-29,31-31,33-33,39-39,41-41,44-44,46-47(Checkpoint
│   │   │   Block)
│   │   │   ├── self_att (SelfAttentionBlock)
│   │   │   │   ├── layernorm_before_attention (LayerNorm) weight:[4096]
│   │   │   │   └── self_attention (Attention)
│   │   │   │       ├── project_q,project_v(Linear) weight:[8388608]
│   │   │   │       │   └── lora (DistributedLowRankLinear) lora_A:[32768] lora_B:[16384]
│   │   │   │       └── project_k,attention_out(Linear) weight:[8388608]
│   │   │   └── ffn (FFNBlock)
│   │   │       ├── layernorm_before_ffn (LayerNorm) weight:[4096]
│   │   │       └── ffn (FeedForward)
│   │   │           ├── w_in (DenseGatedACT)
│   │   │           │   └── w_0,w_1(Linear) weight:[20971520]
│   │   │           └── w_out (Linear) weight:[20971520]
│   │   ├── 1-1,3-3,5-5,11-12,18-18,35-35(CheckpointBlock)
│   │   │   └── ffn (FFNBlock)
│   │   │       ├── layernorm_before_ffn (LayerNorm) weight:[4096]
│   │   │       └── ffn (FeedForward)
│   │   │           ├── w_in (DenseGatedACT)
│   │   │           │   └── w_0,w_1(Linear) weight:[20971520]
│   │   │           └── w_out (Linear) weight:[20971520]
│   │   └── 17-17,21-21(CheckpointBlock)
│   │       └── self_att (SelfAttentionBlock)
│   │           ├── layernorm_before_attention (LayerNorm) weight:[4096]
│   │           └── self_attention (Attention)
│   │               ├── project_q,project_v(Linear) weight:[8388608]
│   │               │   └── lora (DistributedLowRankLinear) lora_A:[32768] lora_B:[16384]
│   │               └── project_k,attention_out(Linear) weight:[8388608]
│   └── output_layernorm (LayerNorm) weight:[4096]
├── input_embedding (EmbeddingExt) weight:[354643968]
└── position_bias (BucketPositionBias) relative_attention_bias:[16384]
[INFO|(OpenDelta)basemodel:696]2023-06-13 20:28:08,940 >> Trainable Ratio: 1867776/2504192000=0.074586%
[INFO|(OpenDelta)basemodel:698]2023-06-13 20:28:08,940 >> Delta Parameter Ratio: 1867776/2504192000=0.074586%
[INFO|(OpenDelta)basemodel:700]2023-06-13 20:28:08,940 >> Static Memory 4.66 GB, Max Memory 9.98 GB
Traceback (most recent call last):
  File "finetune_cpm_bee.py", line 422, in <module>
    main()
  File "finetune_cpm_bee.py", line 418, in main
    finetune(args, tokenizer, model, optimizer, lr_scheduler, optim_manager)
  File "finetune_cpm_bee.py", line 262, in finetune
    logits, _ = model(
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/models/bee.py", line 179, in forward
    hidden_states = self.encoder(hidden_states, attention_mask, position_bias)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/layers/transformer.py", line 106, in forward
    hidden_states = self.layers(hidden_states, attention_mask, position_bias)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/bmtrain/block_layer.py", line 868, in forward
    outputs = OpTransformerBlockList.apply(placeholder, self, self.save_list, hidden_state, *args)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/bmtrain/block_layer.py", line 683, in forward
    hidden_state = self._modules[str(i)]._module._call_impl(hidden_state, *args)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/layers/blocks.py", line 230, in forward
    hidden_states = self.self_att(
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/layers/blocks.py", line 86, in forward
    x = self.self_attention(x, x, attention_mask, position_bias, use_cache, past_key_value)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/layers/attention.py", line 75, in forward
    h_q = self.project_q(hidden_q) / math.sqrt(math.sqrt(self.dim_head))
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/opendelta/utils/decorate.py", line 47, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/opendelta/utils/data_parallel.py", line 36, in parallel_caller
    ret_1 = _org_func(*args, **kwargs)
  File "/home/c2j/workspace/GPT/CPM-Bee_tutorials/src/cpm_live/layers/linear.py", line 55, in forward
    x = F.linear(x, self.weight)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 22086) of binary: /home/c2j/workspace/ENVCPM/bin/python
Traceback (most recent call last):
  File "/home/c2j/workspace/ENVCPM/bin/torchrun", line 10, in <module>
    sys.exit(main())
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/c2j/workspace/ENVCPM/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
finetune_cpm_bee.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-06-13_20:28:14
  host      : Goku
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 22086)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported torchrun command and compare the --max-length 64 and 128 runs. Read finetune_cpm_bee.py around finetune(), then follow the traceback through src/cpm_live/layers/attention.py and linear.py to the failing F.linear call. Done means the 128-token fine-tuning run completes without the CUBLAS error and produces output in results/.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.