google-research / google-research/t5x
Training freezes when trying to evaluate previously saved checkpoint
- Dominant language
- Python
- Stars
- 3k
- Forks
- 338
- PR merge metrics
- No merged PRs in 30d
Description
I'm fine-tuning an mT5 small checkpoint on a custom mixture. For debugging, I've set `eval_period=25` and `utils.SaveCheckpointConfig.period=50` in finetune.gin.
The training job freezes inside `trainer.py` whenever it tries to compute metrics on a saved checkpoint (see attached screenshot)

Contributor guide
Research direction
Start with finetune.gin and the SaveCheckpointConfig.period setting, then inspect trainer.py where metrics are computed for saved checkpoints. Reproduce the freeze using eval_period=25 and checkpoint period=50, and trace that evaluation path. Done means training completes checkpoint evaluation and reports metrics instead of freezing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100