bytedance / bytedance/1d-tokenizer
Eval code not syncing between GPUs
- Dominant language
- Jupyter Notebook
- Stars
- 1.2k
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
looking at [train_utils.py:581](https://github.com/bytedance/1d-tokenizer/blob/377fea5371940f15fa8624cbc62abc5d47ea0b6f/utils/train_utils.py#L581), in the eval_reconstruction section, there doesn't seem to be any sync of eval values between the GPUs. So each GPU computes rFID etc on part of the eval set, but then only the value from main GPU is reported. Not sure if I am looking at it right or if you guys are aware of this and want it this, but wanted to bring it to your attention just in case.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at utils/train_utils.py:581 in the eval_reconstruction section and trace how evaluation values are computed and reported across GPUs. Compare single- and multi-GPU evaluation, then verify that rFID and related metrics represent the full evaluation set rather than only the main GPU's portion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100