AnswerDotAI / AnswerDotAI/RAGatouille

How to index collection using generator function?

Open
#220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

I have a large collection of 16 million passages that I want to index. It's not practical to keep all documents and ids in memory as a list to pass it to the index function. Is there a way to index large collections using generators?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names the index function but no file or test. Start by locating that public entry point and tracing how it handles documents and ids, then inspect existing indexing tests. Done means a large collection can be consumed without requiring all documents and ids as in-memory lists.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.