dmlc / dmlc/dgl

Add versioning to all DGLDatasets to force reloading when codes are changed

Open
#4,293 1 comment 0 reactions 1 assignee Claimed by @mufeili View on GitHub
feature request
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Add versioning to all DGLDatasets to detect:
* Changes to the raw dataset files stored in S3. If so, re-download the dataset.
* Changes to the preprocessing code. If so, ignore the previous local cache and reprocess the data.

## Motivation

Brought up by #3987 which asks to revert the default reordering behavior of DGL builtin datasets. The issue is that even if we've implemented the request, users may still load cached datasets from local disk which may not reflect the latest change. Therefore, we require some versioning mechanism to detect those changes.

cc @mufeili

## Alternatives

Use a different dataset folder whenever DGL updates. This could cause excessive disk storage use.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.