dmlc / dmlc/dgl

Fused Neigbhor Sampling reproducibility issue

Open
#7,831 3 comments 0 reactions 0 assignees View on GitHub
stale-issue
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

Setting the seed and repeating the fused neighborhood sampling for a source code does not reproduce the same subgraph, have identified a fix that will be slower but allow reproducible subgraphs

## To Reproduce

Steps to reproduce the behavior:

1. Set OMP_NUM_THREADS to >= 2
2. Define NeighborSampler with fused=True
3. Set dgl.seed, dgl.random.seed before calling dgl.dataloading.NeighborSampler.sample_blocks
4. Repeat step 3. and compare blocks[0].srcdata['feat']

The following change allowed for the above to have the same output

```
diff --git a/src/graph/sampling/neighbor/neighbor.cc b/src/graph/sampling/neighbor/neighbor.cc
index 5393a200..471600dd 100644
--- a/src/graph/sampling/neighbor/neighbor.cc
+++ b/src/graph/sampling/neighbor/neighbor.cc
@@ -14,6 +14,7 @@

#include
#include
+#include

#include "../../../array/cpu/concurrent_id_hash_map.h"
#include "../../../c_api_common.h"
@@ -488,6 +489,7 @@ SampleNeighborsFused(
}
}
IdType offset = new_nodes_vec[lhs_node_type].size();
+ auto length = offset;
new_nodes_vec[lhs_node_type].resize(global_prefix_col.back());
for (int thread_id = 0; thread_id < num_threads_col; ++thread_id) {
memcpy(
@@ -496,6 +498,7 @@ SampleNeighborsFused(
src_nodes_local[thread_id].size() * sizeof(IdType));
offset += src_nodes_local[thread_id].size();
}
+ std::sort(std::execution::par_unseq, new_nodes_vec[lhs_node_type].begin() + length, new_nodes_vec[lhs_node_type].end());
}
}
```

This results is the sampling being slower but reproducible, would there be any alternative for multithreaded fused sampling being reproducible?

## Environment

- DGL Version (e.g., 1.0): 2.4
- Backend Library & Version : Torch 2.3.1
- OS (e.g., Linux): Linux
- How you installed DGL (`conda`, `pip`, source): source
- Build command you used (if compiling from source): cmake -DBUILD_TYPE='release' -DUSE_LIBXSMM=ON -DUSE_OPENMP=ON -DBUILD_SPARSE=OFF -DUSE_EPOLL=OFF ..
- Python version: 3.9
- CPU: Intel Xeon

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with OMP_NUM_THREADS >= 2 and fused NeighborSampler, then read SampleNeighborsFused in src/graph/sampling/neighbor/neighbor.cc. Compare repeated blocks[0].srcdata['feat'] results with the reported ordering change. Done means repeated sampling with the same seeds produces identical subgraphs without an unacceptable regression in sampling speed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.