Support Sparse Recommendation Model Training
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
## Background
Graph learning is one of the major workloads supported by GraphScope. GNN's main application scenarios include recommendation systems, financial risk control, and biomedical development. Among them, GLE (the learning engine in GraphScope) has been widely applied in the scenario of recommendation. Most of the recommendation models are sparse model, where the size of embedding can reach the scale of tens of billions. To further integrate with the Pytorch ecosystem, we have launched [grephlearn-for-pytorch](https://github.com/alibaba/graphlearn-for-pytorch), a graph learning library for PyTorch that makes distributed GNN training and inference easy and efficient. When utilizing GPU for model training, grephlearn-for-pytorch can outperform GLE on end-to-end tasks by a significant margin. Additionally, the process of developing models is much easier compared to GLE, which is based on TensorFlow 1.x. We plan to build the next generation of GLE engine based on grephlearn-for-pytorch in GraphScope, thus supporting large-scale sparse model training on grephlearn-for-pytorch is very important.
[TorchRec](https://github.com/pytorch/torchrec) is a PyTorch library for recommendation systems. When building recommendation systems, it is often necessary to use embeddings to represent entities such as products or pages. As the number of entities grows, the size of the embedding table may exceed the memory of a single GPU. The common approach is to shard the embedding table, which is a model parallel method. TorchRec encapsulates a series of APIs to support model parallel training based on embedding sharding (DMP). In this project, we will implement the integration of grephlearn-for-pytorch with TorchRec and implement a distributed embedding module based on GraphSAGE for the recommendation GNN model.
Tasks:
- Support large-scale embedding in grephlearn-for-pytorch based on TorchRec.
- Properly design the APIs, provide a simple and user-friendly distributed solution for recommendation models.
Contributor guide
Research direction
Start by reviewing the grephlearn-for-pytorch and TorchRec integration requirements, focusing on embedding sharding and the proposed GraphSAGE-based distributed embedding module. Done means supporting large-scale embeddings and providing simple, user-friendly APIs for distributed recommendation model training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100