AnswerDotAI / AnswerDotAI/RAGatouille

Couldn't use multiple GPU to search

Open
#177 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Update: I found I can use multiple GPUs to index but couldn't use multiple GPUs to search. Is this a known limitation?

I got CUDA OOM error when I index too many documents and when chunk size is too large. Are there ways to limit GPU memory usage? Thank you!

When I have 4 GPUs RAGatouille seems to only use 1. I set n_gpu=4 and it still only uses 1.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by reproducing the report with n_gpu=4, comparing indexing and search behavior, and checking the CUDA out-of-memory case with large document counts or chunk sizes. Done means documenting whether multi-GPU search is supported and identifying the relevant GPU memory limitation or required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.