EleutherAI / EleutherAI/bergson
Checkpoint Hessian fit
- Dominant language
- Python
- Stars
- 86
- Forks
- 29
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 37
Description
The KFAC fit is a large accumulation but doesn't save progress checkpoints.
Every N documents, rank 0 should write the accumulator tensors + doc cursor to `.part/fit_state.pt`; load it in on resume and skip the stream to the cursor. The same thing should happen in the correction pass.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the KFAC fit and correction pass entry points, then inspect how accumulator tensors and the document stream cursor are maintained. The work is done when rank 0 checkpoints both passes to .part/fit_state.pt every N documents and resume restores the tensors and skips the stream to the saved cursor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100