deepseek-ai / deepseek-ai/DeepSeek-Coder
训练数据切分问题
Open
- Dominant language
- Python
- Stars
- 24.3k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
对于训练代码文件超过context大小时，需要切分成不同个的4096 minibatch。想问一下，咱们这个切分是按照token级别切分，还是按行级别切分，或者是function级别切分？
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the training-data preprocessing entry point; the issue does not name a file or test. Determine whether code exceeding the context size is split into 4096 minibatches by token, line, or function boundaries, then document the confirmed behavior and its expected boundary handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100