使用单卡进行微调时，dataset会报End of dataset的错
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 176
- PR merge metrics
- No merged PRs in 30d
Description
如题，使用预处理数据脚本预处理数据后，在进行微调时，dataset报错，不太确定是预处理脚本的问题，还是dataset迭代的问题
376 if self._max_repeat_times is not None:
377 if self._repeat_times >= self._max_repeat_times:
--> 378 raise EOFError("End of dataset")
379 print('_prepare_new_epoch, 2')
380 nw_unused_block: List[int] = []
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the preprocessing data script and the dataset iteration code around lines 376-380, where EOFError("End of dataset") is raised during single-card fine-tuning. Reproduce the failure with the preprocessed data, then determine whether the data preparation or dataset iteration causes it. Done means single-card fine-tuning no longer ends unexpectedly, with the cause identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100