deepspeedai / deepspeedai/DeepSpeed
Using FAISS along deepspeed for online negative mining
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I want to update my dataset periodically (lets say every 10k steps) with new sampling strategy.
Ideally I would utilize all GPUs to calculate all image embeddings, free GPU memory and let FAISS quickly calculate neighbours for each sample. The neighbour indices get saved to the file and dataloader now constructs batches using updated neighbours.
How would I do that with DeepSpeed? Is it possible to:
- pause training and use the model to quickly calculate embeddings and update dataloader
- let faiss use all gpus to get nearest neighbours and resume training (deallocate memory eaten by DeepSpeed but not currently used to be more exact)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file, test, or entry point is named. Start by reviewing DeepSpeed's training-loop and dataloader integration points, then determine how the proposed pause, embedding extraction, GPU FAISS neighbor calculation, and resume flow would fit. Done would require a maintainer-confirmed design or a narrower issue with code locations and acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100