microsoft / microsoft/Graphormer
can not reproduce PCQM4M-LSC results
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 374
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
I tried to reproduce the results of Graphormer_SMALL on the PCQM4M-LSC dataset using the v1.0 branch.
[ -z "${exp_name}" ] && exp_name="pcq"
[ -z "${seed}" ] && seed="1"
[ -z "${arch}" ] && arch="--ffn_dim 512 --hidden_dim 512 --weight_decay 0.0 --intput_dropout_rate 0.0 --dropout_rate 0.1 --n_layers 6 --peak_lr 3e-4 --edge_type multi_hop --multi_hop_max_dist 5"
[ -z "${batch_size}" ] && batch_size="256"
echo -e "\n\n"
echo "=====================================ARGS======================================"
echo "arg0: $0"
echo "exp_name: ${exp_name}"
echo "arch: ${arch}"
echo "seed: ${seed}"
echo "batch_size: ${batch_size}"
echo "==============================================================================="
default_root_dir="../../exps/pcq/$exp_name/$seed"
mkdir -p $default_root_dir
n_gpu=$(nvidia-smi -L | wc -l)
python ../../graphormer/entry.py --num_workers 8 --seed $seed --batch_size $batch_size \
--dataset_name PCQM4M-LSC \
--gpus $n_gpu --accelerator ddp --precision 16 --gradient_clip_val 5.0 \
$arch \
--default_root_dir $default_root_dir
My machine also has 8 GPUs. In this case, is the equivalent batch size 256 * 8 = 2048? After training, according to Tensorboard curves, the valid_mae is about 0.135, while it is 0.1264 for Graphormer_SMALL as reported in the paper. Do you have any ideas about the performance gap? It is suggested to set batch size 1024, so should I change the batch size to 128 (1024/8) for an 8-GPU machine?
Do you run baseline methods in Table 1 to report their performance?
I will be very appreciated if you can answer these questions!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the v1.0 branch and inspect graphormer/entry.py together with the posted training command, focusing on batch-size and multi-GPU arguments. Reproduce the PCQM4M-LSC Graphormer_SMALL run and compare its TensorBoard valid_mae with the reported 0.1264 result. Done means documenting the effective configuration and explaining the performance gap or identifying the missing reproduction step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100