dmlc / dmlc/dgl

Proper link prediction evaluation in DistDGL

Open
#3,706 1 comment 0 reactions 1 assignee Claimed by @BarclayII View on GitHub
feature request topic: Distributed DGL
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

DistDGL does not allow two graphs for inference. In link prediction this will create problem of either not correct evaluation if testing edges are included in the graph or a lot of custom code to do it properly. I have some ideas I can share with the team.

**Main concern**
Imagine you train a GNN and the testing edges exists in the graph. Naturally, you will see the performance in the training set improving as the model trains. You will also see the performance in the testing edge improve and at a certain epoch K it will start degrading. The problem is that the overfitting will happen much earlier than epoch K, but since the testing edges exist in the graph we will not be able to see it. This will lead to a huge problem in production where the inference happens usually in unobserved edges.

The main solution here is to support a testing and a training graph in the DistDGL as we do in DGL.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.