microsoft / microsoft/Graphormer

Evaluation gets nan mae

Open
#161 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
374
PR merge metrics
No merged PRs in 30d

Description

I want to evaluate the performance of pretrained Graphormer using testing data and I use following code

python evaluate.py \
    --user-dir ../../graphormer \
    --num-workers 16 \
    --ddp-backend=legacy_ddp \
    --dataset-name pcqm4m \
    --dataset-source ogb \
    --task graph_prediction \
    --criterion l1_loss \
    --arch graphormer_base \
    --num-classes 1 \
    --batch-size 64 \
    --pretrained-model-name pcqm4mv1_graphormer_base \
    --load-pretrained-model-output-layer \
    --split test \
    --seed 1

It is similar to code in https://graphormer.readthedocs.io/en/latest/Quick-Start.html#evaluate-pre-trained-models, and the only difference is --split.

However, I get result like this. I don't know why.
图片

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with evaluate.py and the documented Quick-Start evaluation command, then compare the effects of using --split test instead of the documented split. Run the provided command and trace where the MAE is computed or reported; done means identifying why the test evaluation produces nan and documenting or correcting that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.