dmlc / dmlc/dgl

Results of seal_ogbl not reproducible

Open
#6,815 1 comment 0 reactions 1 assignee Claimed by @rudongyu View on GitHub
stale-issue
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

As is reported in the readme at `examples/pytorch/ogb/seal_ogbl`, the test accuracy on ogbl-collab (Hits@50) and ogbl-citation2 (MRRd) is 64.23%±0.57% and 86.29%±0.47%, respectively.

However, after I tried on these two dataset with `main.py`, the results are far from that reported:
* For ogbl-collab (Hits@50), I get the highest validation accuracy 64.22% while the highest test accuracy is only 38.91%.
* For ogbl-citation2 (MRRd), I get the highest validation accuracy 65.65% while the highest test accuracy is only 62.65%.

I followed the command line arguments given in the readme to run the script.

The training log can be found here:
[log_collab.txt](https://github.com/dmlc/dgl/files/13760736/log_collab.txt)
[log_citation2.txt](https://github.com/dmlc/dgl/files/13760737/log_citation2.txt)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.