Results of seal_ogbl not reproducible
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
As is reported in the readme at `examples/pytorch/ogb/seal_ogbl`, the test accuracy on ogbl-collab (Hits@50) and ogbl-citation2 (MRRd) is 64.23%±0.57% and 86.29%±0.47%, respectively.
However, after I tried on these two dataset with `main.py`, the results are far from that reported:
* For ogbl-collab (Hits@50), I get the highest validation accuracy 64.22% while the highest test accuracy is only 38.91%.
* For ogbl-citation2 (MRRd), I get the highest validation accuracy 65.65% while the highest test accuracy is only 62.65%.
I followed the command line arguments given in the readme to run the script.
The training log can be found here:
[log_collab.txt](https://github.com/dmlc/dgl/files/13760736/log_collab.txt)
[log_citation2.txt](https://github.com/dmlc/dgl/files/13760737/log_citation2.txt)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.