bytedance / bytedance/1d-tokenizer

[TATiTok] The self-test results of the reconstruction metrics are inconsistent with the paper. The measured IS is 20 points lower. Could you please help figure out what's wrong with my test?

Open
#116 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Image

I used the following script to test the reconstruction metrics. The IS is much lower. Also, the PSNR is not reported in the paper. I want to confirm if it is similar to the 24.8 I measured?

## Test Script:
[eval_resconstruct0-1 (1).py](https://github.com/user-attachments/files/23785057/eval_resconstruct0-1.1.py)
[evaluator.py](https://github.com/user-attachments/files/23784742/evaluator.py)

## Test cmd:
```bash
python -m accelerate.commands.launch \
--num_processes=$(nvidia-smi --list-gpus | wc -l) \
--num_machines=$NUM_MACHINES \
--machine_rank=$MACHINE_RANK \
--main_process_ip=$MAIN_PROCESS_IP \
--main_process_port=$MAIN_PROCESS_PORT \
--mixed_precision=bf16 \
eval_resconstruct0-1py \
--eval_data_dir imagenet/CLS-LOC/val \
--model_name turkeyju/tokenizer_tatitok_sl128_vae \
--class_file imagenet_1k_class.txt \
--batch_size 64 \
--model_image_size 256 \
--eval_image_size 256 \
--buffer_size 8 \
--enable_rfid \
--enable_inception_score
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the attached eval_resconstruct0-1 (1).py and evaluator.py scripts, then reproduce the reported command with the specified dataset, model, and evaluation settings. Compare the measured inception score and PSNR with the paper and determine whether the discrepancy comes from the test setup or metric evaluation; done means the cause and expected values are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.