AnswerDotAI / AnswerDotAI/RAGatouille
How to index collection using generator function?
Open
- Dominant language
- Python
- Stars
- 4k
- Forks
- 276
- PR merge metrics
- No merged PRs in 30d
Description
I have a large collection of 16 million passages that I want to index. It's not practical to keep all documents and ids in memory as a list to pass it to the index function. Is there a way to index large collections using generators?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names the index function but no file or test. Start by locating that public entry point and tracing how it handles documents and ids, then inspect existing indexing tests. Done means a large collection can be consumed without requiring all documents and ids as in-memory lists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100