microsoft / microsoft/Graphormer

can not reproduce PCQM4M-LSC results

Open
#103 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
374
PR merge metrics
No merged PRs in 30d

Description

Hi there,

I tried to reproduce the results of Graphormer_SMALL on the PCQM4M-LSC dataset using the v1.0 branch.

[ -z "${exp_name}" ] && exp_name="pcq"
[ -z "${seed}" ] && seed="1"
[ -z "${arch}" ] && arch="--ffn_dim 512 --hidden_dim 512 --weight_decay 0.0 --intput_dropout_rate 0.0 --dropout_rate 0.1 --n_layers 6 --peak_lr 3e-4 --edge_type multi_hop --multi_hop_max_dist 5"
[ -z "${batch_size}" ] && batch_size="256"

echo -e "\n\n"
echo "=====================================ARGS======================================"
echo "arg0: $0"
echo "exp_name: ${exp_name}"
echo "arch: ${arch}"
echo "seed: ${seed}"
echo "batch_size: ${batch_size}"
echo "==============================================================================="

default_root_dir="../../exps/pcq/$exp_name/$seed"
mkdir -p $default_root_dir
n_gpu=$(nvidia-smi -L | wc -l)

python ../../graphormer/entry.py --num_workers 8 --seed $seed --batch_size $batch_size \
      --dataset_name PCQM4M-LSC \
      --gpus $n_gpu --accelerator ddp --precision 16 --gradient_clip_val 5.0 \
      $arch \
      --default_root_dir $default_root_dir

My machine also has 8 GPUs. In this case, is the equivalent batch size 256 * 8 = 2048? After training, according to Tensorboard curves, the valid_mae is about 0.135, while it is 0.1264 for Graphormer_SMALL as reported in the paper. Do you have any ideas about the performance gap? It is suggested to set batch size 1024, so should I change the batch size to 128 (1024/8) for an 8-GPU machine?

Do you run baseline methods in Table 1 to report their performance?

I will be very appreciated if you can answer these questions!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the v1.0 branch and inspect graphormer/entry.py together with the posted training command, focusing on batch-size and multi-GPU arguments. Reproduce the PCQM4M-LSC Graphormer_SMALL run and compare its TensorBoard valid_mae with the reported 0.1264 result. Done means documenting the effective configuration and explaining the performance gap or identifying the missing reproduction step.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.