tensorflow / tensorflow/recommenders
[Question] How to train/ continuous train with large dataset?
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
The tutorial example illustrates how we recommend movies given to users. I have two follow-up questions regarding when this algorithm scale-up:
- What if the number of users increases to 50M and the number of movies to 100k.(can't feed everything into memory..)
- What to do when new users and new movies are added in, How would this architecture handle continuous training?
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tutorial example referenced in the issue and examine how it handles users, movies, and training data. The issue names no files or tests; done would be clear guidance covering large datasets and the addition of new users and movies during continued training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data-engineering, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100