google-research / google-research/t5x

Training freezes when trying to evaluate previously saved checkpoint

Open
#344 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3k
Forks
338
PR merge metrics
No merged PRs in 30d

Description

I'm fine-tuning an mT5 small checkpoint on a custom mixture. For debugging, I've set `eval_period=25` and `utils.SaveCheckpointConfig.period=50` in finetune.gin.
The training job freezes inside `trainer.py` whenever it tries to compute metrics on a saved checkpoint (see attached screenshot)

![Screenshot 2022-03-14 11 19 52 PM](https://user-images.githubusercontent.com/7523108/158232459-037e80fa-c7c6-4fd3-a629-b1a5bdf53c83.png)

Contributor guide

Open the contributing guide

Research direction

Start with finetune.gin and the SaveCheckpointConfig.period setting, then inspect trainer.py where metrics are computed for saved checkpoints. Reproduce the freeze using eval_period=25 and checkpoint period=50, and trace that evaluation path. Done means training completes checkpoint evaluation and reports metrics instead of freezing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.