Graphsage算法模型训练后如何导出embedding表?
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
背景:9千万个点,20亿边,分布式训练后(1ps + 20worker)得到cpkt模型,然后进行embedding表导出,执行如何命令:
python -m tf_euler --data_dir hdfs://path/data --model_dir model --max_id 90000000 --feature_idx 0 --feature_dim 1 --model graphsage --mode save_embedding
由于save_embedding不像train那样实现分布式计算,所以执行上面命令后会加载所有数据到单台机器内存后,导致内存不足,从而无法导出embedding表。所以,请问对于大规模的图,怎么导出模型训练后对应的embedding表?
@yangsiran
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `python -m tf_euler` `save_embedding` mode and compare it with the distributed `train` mode described in the issue. Investigate how the 90-million-node export loads data and identify how the embedding table could be exported without requiring one machine to hold all data. Done means the trained GraphSAGE embeddings can be exported for the stated large graph without a single-machine out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100