alibaba / alibaba/euler

Graphsage算法模型训练后如何导出embedding表?

Open
#101 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

背景:9千万个点,20亿边,分布式训练后(1ps + 20worker)得到cpkt模型,然后进行embedding表导出,执行如何命令:

python -m tf_euler --data_dir hdfs://path/data --model_dir model --max_id 90000000 --feature_idx 0 --feature_dim 1 --model graphsage --mode save_embedding

由于save_embedding不像train那样实现分布式计算,所以执行上面命令后会加载所有数据到单台机器内存后,导致内存不足,从而无法导出embedding表。所以,请问对于大规模的图,怎么导出模型训练后对应的embedding表?

@yangsiran

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `python -m tf_euler` `save_embedding` mode and compare it with the distributed `train` mode described in the issue. Investigate how the 90-million-node export loads data and identify how the embedding table could be exported without requiring one machine to hold all data. Done means the trained GraphSAGE embeddings can be exported for the stated large graph without a single-machine out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.