modelscope / modelscope/easydistill
Handling Long run distillation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 475
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
when we run distillation pipeline with huge data, there is a chance that process fails in infer phase due to any network like issues. In such cases we need restart mechanism from where we left off. But in your case when you used 1m instruction dataset, didn't you have any issues??
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the distillation pipeline's infer phase and the code that processes the instruction dataset; the issue does not name specific files or tests. Reproduce or inspect failure handling with a large dataset and network interruption, then define and verify restart behavior that resumes from the last completed work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100