alibaba / alibaba/euler

在大规模数据训练时,采用euler_ops.get_dense_feature OP存在大量获取不到label信息的问题

Open
#128 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

如题,数据规模为4000万级节点10亿级边,采用8机分布式训练,单机内存为120G。 训练过程中,打印每个batch的节点的label信息,发现大部分的节点label为全0(查询节点label信息失败?)(已在单机加载单个.dat文件验证过节点label是存在的, 小规模的数据分布式训练验证是正常的)
调用的方法是:
euler_ops.get_dense_feature(batch, [1], [4])
graph初始化方法:
```
euler_ops.initialize_shared_graph(
directory=flags_obj.data_dir,
zk_addr=flags_obj.euler_zk_addr,
zk_path=flags_obj.euler_zk_path,
shard_idx=flags_obj.task_index,
shard_num=len(flags_obj.worker_hosts),
global_sampler_type='node')
```
json格式数据样例( float_feature: 0是节点feature, 1是节点label):
```
{"node_weight": 1.0, "uint64_feature": {}, "float_feature": {"1": [1, 0], "0": [0.01988, 0.004274, 0.485382, 0.489668, 0.0, 0.0, 0.0, 0.0, 0.0, 0.000796, 0.0, 0.0, 0.0, 0.0, 0.0]}, "edge": [{"src_id": 100001423, "weight": 0.223607, "uint64_feature": {}, "float_feature": {}, "dst_id": 301156921, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 100001423, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 324544525, "edge_type": 0, "binary_feature": {}}, {"src_id": 100001423, "weight": 0.25, "uint64_feature": {}, "float_feature": {}, "dst_id": 7918006, "edge_type": 0, "binary_feature": {}}], "node_type": 0, "node_id": 100001423, "neighbor": {"1": {}, "0": {"301156921": 0.223607, "7918006": 0.25, "324544525": 0.25, "100001423": 0.25}}, "binary_feature": {}}
```
训练过程中打印的batch label信息(每行代表一个节点的label,至少有一个1才是正常的):
```
[[0. 1.]
[1. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 1.]
[0. 0.]
[0. 0.]]
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with euler_ops.get_dense_feature and the initialize_shared_graph configuration shown in the report, comparing the large distributed run with the verified single-machine and small distributed cases. Trace how node labels are retrieved across shards; done means large-scale training returns the expected nonzero labels instead of mostly all-zero values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.