dmlc / dmlc/dgl

Get stuck when trying to generation random bipartite with 2B edges

Open
#4,288 3 comments 0 reactions 1 assignee Assigned to @peizhou001 View on GitHub
bug:confirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

I tried with below code to generate random bipartite. It succeeded with 1B edges( takes about 800 seconds) but get stuck with 2B edges(more than a few hours). No error/exception is thrown. I tried to dig in and find that below `insert` always failed since `selected.size() > 1073500*1000`. `1073500*1000` is a rough number, not deterministic, but always failed around it. Seems `RandInt()` always returns duplicate value.
https://github.com/dmlc/dgl/blob/6e1be69a84ba3e17e8e4db3c3768448f3620ecf4/src/random/cpu/choice.cc#L95-L99

## To Reproduce

Steps to reproduce the behavior:

machine: `x2idn.16xlarge`, 1T CPU RAM, 64 CPUs.

```
num_nodes = 5 * 1000 * 1000
num_edges = 2 * 1000 * 1000 * 1000
num_src_nodes = num_nodes//2
num_dst_nodes = num_nodes - num_src_nodes
tic = time.time()
t_tic = tic
g = dgl.rand_bipartite('node1', 'edge', 'node2',
num_src_nodes, num_dst_nodes, num_edges)
```

## Expected behavior
generation should finish in linear increase of 1B edges cases, or error/exception should be thrown.

## Environment

- DGL Version (e.g., 1.0): master, 0.9.x
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
- OS (e.g., Linux): Linux
- How you installed DGL (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:

## Additional context

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.