在大规模数据训练时,采用euler_ops.get_dense_feature OP存在大量获取不到label信息的问题
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
如题,数据规模为4000万级节点10亿级边,采用8机分布式训练,单机内存为120G。 训练过程中,打印每个batch的节点的label信息,发现大部分的节点label为全0(查询节点label信息失败?)(已在单机加载单个.dat文件验证过节点label是存在的, 小规模的数据分布式训练验证是正常的)
调用的方法是:
euler_ops.get_dense_feature(batch, [1], [4])
graph初始化方法:
```
euler_ops.initialize_shared_graph(
directory=flags_obj.data_dir,
zk_addr=flags_obj.euler_zk_addr,
zk_path=flags_obj.euler_zk_path,
shard_idx=flags_obj.task_index,
shard_num=len(flags_obj.worker_hosts),
global_sampler_type='node')
```
json格式数据样例( float_feature: 0是节点feature, 1是节点label):
```
{"node_weight": 1.0, "uint64_feature": {}, "float_feature": {"1": [1, 0], "0": [0.01988, 0.004274, 0.485382, 0.489668, 0.0, 0.0, 0.0, 0.0, 0.0, 0.000796, 0.0, 0.0, 0.0, 0.0, 0.0]}, "edge": [{"src_id": 100001423, "weight": 0.223607, "uint64_feature": {}, "float_feature": {}, "dst_id": 301156921, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 100001423, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 324544525, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 7918006, "edge_type": 0, "binary_feature": {}}], "node_type": 0, "node_id": 100001423, "neighbor": {"1": {}, "0": {"301156921": 0.223607, "7918006": 0.25, "324544525": 0.25, "100001423": 0.25}}, "binary_feature": {}}
```
训练过程中打印的batch label信息(每行代表一个节点的label,至少有一个1才是正常的):
```
[[0. 1.]
[1. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 1.]
[0. 0.]
[0. 0.]]
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with euler_ops.get_dense_feature and the initialize_shared_graph configuration shown in the report, comparing the large distributed run with the verified single-machine and small distributed cases. Trace how node labels are retrieved across shards; done means large-scale training returns the expected nonzero labels instead of mostly all-zero values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100