bytedance / bytedance/1d-tokenizer

Eval code not syncing between GPUs

Open
#83 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Hello,

looking at [train_utils.py:581](https://github.com/bytedance/1d-tokenizer/blob/377fea5371940f15fa8624cbc62abc5d47ea0b6f/utils/train_utils.py#L581), in the eval_reconstruction section, there doesn't seem to be any sync of eval values between the GPUs. So each GPU computes rFID etc on part of the eval set, but then only the value from main GPU is reported. Not sure if I am looking at it right or if you guys are aware of this and want it this, but wanted to bring it to your attention just in case.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at utils/train_utils.py:581 in the eval_reconstruction section and trace how evaluation values are computed and reported across GPUs. Compare single- and multi-GPU evaluation, then verify that rFID and related metrics represent the full evaluation set rather than only the main GPU's portion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.