微调Qwen2，use_varlen_attn=True时报错
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
use_varlen_attn=False时正常，设为True时报如下错误：
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:219, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
请问怎么回事啊？另外，同样的环境，微调其它模型，将use_varlen_attn=True不会报错
我尝试更新pytorch版本（附带更新nccl版本），结果又报了如下错误：
flash_attn_2_cuda.cpython-39-x86_64-linux-gnu.so: undefined symbol:
Contributor guide
Assessment
This issue has not been assessed yet.