microsoft / microsoft/SynapseML
[Lightgbm] Support saving checkpoints during training
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Is your feature request related to a problem? Please describe.**
Hi, @imatiach-msft , I'm using mmlspark lightgbm to train a ranking model that costs about 9 to 10 hours on a large dataset. My spark is deployed on yarn, there are several times the training job is failed caused by the lost node on my yarn cluster.
Can mmlspark lightgbm support saving checkpoints during training so that we can continue training next time?
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the mmlspark LightGBM training entry point and reviewing how the ranking model runs on Spark/YARN. Clarify the checkpoint and resume requirements, including what training state must persist; done means a failed training job can continue from a saved checkpoint and the behavior is verified for this workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100