bytedance / bytedance/1d-tokenizer
[TATiTok] The self-test results of the reconstruction metrics are inconsistent with the paper. The measured IS is 20 points lower. Could you please help figure out what's wrong with my test?
- Dominant language
- Jupyter Notebook
- Stars
- 1.2k
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
I used the following script to test the reconstruction metrics. The IS is much lower. Also, the PSNR is not reported in the paper. I want to confirm if it is similar to the 24.8 I measured?
## Test Script:
[eval_resconstruct0-1 (1).py](https://github.com/user-attachments/files/23785057/eval_resconstruct0-1.1.py)
[evaluator.py](https://github.com/user-attachments/files/23784742/evaluator.py)
## Test cmd:
```bash
python -m accelerate.commands.launch \
--num_processes=$(nvidia-smi --list-gpus | wc -l) \
--num_machines=$NUM_MACHINES \
--machine_rank=$MACHINE_RANK \
--main_process_ip=$MAIN_PROCESS_IP \
--main_process_port=$MAIN_PROCESS_PORT \
--mixed_precision=bf16 \
eval_resconstruct0-1py \
--eval_data_dir imagenet/CLS-LOC/val \
--model_name turkeyju/tokenizer_tatitok_sl128_vae \
--class_file imagenet_1k_class.txt \
--batch_size 64 \
--model_image_size 256 \
--eval_image_size 256 \
--buffer_size 8 \
--enable_rfid \
--enable_inception_score
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the attached eval_resconstruct0-1 (1).py and evaluator.py scripts, then reproduce the reported command with the specified dataset, model, and evaluation settings. Compare the measured inception score and PSNR with the paper and determine whether the discrepancy comes from the test setup or metric evaluation; done means the cause and expected values are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100