dmlc / dmlc/dgl

[GraphBolt] Dataset dtypes

Open
#7,130 19 comments 0 reactions 1 assignee Claimed by @pyynb View on GitHub
Work Item
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🔨Work Item

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description

Basically, any tensor that has indices corresponding to node ids should be stored with the int32 dtype.

- [ ] ogbn-arxiv dataset has the int64 dtypes while it can use int32 for csc_indptr and indices tensors and all the rest of the node ids tensors.
- [x] ogbn-papers100M has less than 2B nodes. Thus, we should store its indices array with the int32 dtype. Also, the train_set, validation_set and test_set should be stored with the int32 dtype as well.
- [x] mag240M dataset needs `indices`, `node_type_offset`, `train_set`, `validation_set`, `test_set` casted into int32. `type_per_edge` into uint8 or int8.

We don't need to do anything for `all_nodes_set` because it automatically gets its dtype from graph.indices.

### Saving features

When saving any feature tensors, we should make sure to use `gb.numpy_save_aligned` instead of `numpy.save`.

- [x] ogbn-papers100M (Thanks to @Liu-rj)
- [ ] ogb-lsc-mag240M

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.