ByteDance-Seed / ByteDance-Seed/Bagel
expected_num_tokens 如果设置成默认的32k的话会OOM
Open
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
expected_num_tokens 如果设置成默认的32k的话会OOM，但是我如果提高num_shard的话通信开销会非常高，所以我只能降低这个长度到12K来保证训练，论文中也提到PT阶段使用的是32K～36K的序列长度，如果我使用缩短长度的话会对最后结果产生很大影响吗
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is identified. Start by reproducing training with expected_num_tokens at 32K and 12K, recording OOM behavior, num_shard communication overhead, and final-result differences; done would require a supported configuration or documented guidance for the trade-off.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100