dmlc / dmlc/dgl

[GraphBolt] items are not shuffled across the whole set if `num_workers>0`

Open
#6,947 1 comment 1 reaction 1 assignee Claimed by @RamonZhou View on GitHub
Work Item
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🔨Work Item

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description
split item before shuffle results in significant accuracy drop. We should shuffle across the whole set first, then split items among workers. It's worth checking if `torch.DL` shuffle in this way.

`buffer_size` of `ItemShufflerAndBatcher` could always be `len(item_set)` to simplify the code logic.

## Depending work items or issues

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.