ml-explore / ml-explore/mlx-examples
[Feature] Continue fine-tuning after the training has been interrupted
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
Feature request as discussed with @awni in discussion 997.
I often had the problem that I had to interrupt the training and then had to start the training all over again. With --resume-adapter-file the training for an existing adapter can be continued. The problem is that the training is not continued where it stopped before. You can reduce the steps accordingly and shuffle the data set. The problem is that I use cosine_decay as learning-rate scheduler with certain values for warmup (309 steps), warmup-init (0) etc. This means that the training starts again at the initial values and do not follow the intended curve. It would be a cool feature if it were possible to take the number of steps already performed from the name of the adapter file or pass it along. MLX could then calculate the continuation point and continue the training with the appropriate learning rate and record from the training set.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with discussion 997 and the training path that handles --resume-adapter-file. Trace how cosine_decay, warmup settings, completed steps, and training-data position are currently handled; done means an interrupted run can resume with the intended scheduler curve and dataset position.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100