modelscope / modelscope/twinkle
twinkle数据集预处理的时间很长
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 258
- Forks
- 50
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 11
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new feature request. / 我已经搜索过现有的 issues，确认这是一个新的 Feature Request。
Feature Request Description / Feature Request 描述
用twinkle训练，数据集预处理的时间很长。我有22万条训练数据，开启了8个进程做数据预处理，显示处理速度是每秒处理3条多数据样本，总处理时间大概在30小时左右，比ms-swift慢不少。
Pull Request / Pull Request 信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing dataset preprocessing with the reported 220,000 samples and 8 processes, recording throughput and total time. Compare the result with ms-swift as the issue requests, and consider the work complete when preprocessing is measurably faster than the reported three samples per second and the total runtime is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100