InternLM / InternLM/xtuner

关于qwen1.5长序列训练的疑问

Open
#761 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.2k
Forks
448
Avg merge
3d 15h
Merged PRs (30d)
26

Description

使用A100-32卡,xtuner训练qwen1.5-32b长序列,有下面几个疑问想请教一下:
1. qwen1.5貌似不支持use_varlen_attn=True参数,这个功能后面有计划增加吗?
2. 对于特定模型来说,use_varlen_attn=True与use_varlen_attn=False,是否有明显差别?
3. 在scheduler设置为按照step更新时,有没有公式可以计算总共需要训练多少个steps?然后好设置begin和end?
4. 在多机多卡训练时,一条pack到max_length的数据,是怎么根据sequence_parallel_size与总卡数来切分数据的?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.