EleutherAI / EleutherAI/bergson

Checkpoint Hessian fit

Open
#453 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
86
Forks
29
Avg merge
1d 22h
Merged PRs (30d)
37

Description

The KFAC fit is a large accumulation but doesn't save progress checkpoints.

Every N documents, rank 0 should write the accumulator tensors + doc cursor to `.part/fit_state.pt`; load it in on resume and skip the stream to the cursor. The same thing should happen in the correction pass.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the KFAC fit and correction pass entry points, then inspect how accumulator tensors and the document stream cursor are maintained. The work is done when rank 0 checkpoints both passes to .part/fit_state.pt every N documents and resume restores the tensors and skips the stream to the saved cursor.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.