NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: Update Ingestor class so that it only keeps a configurable amount of results in memory and spills the rest to disk
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Currently preventing usage
Please provide a clear description of problem this feature solves
The Ingestor class's ingest method accumulates the results of all processed jobs in memory while waiting for jobs to complete. For very large data sets this can result in GBs of memory that are held until the entire job has completed. We should modify this behavior so that after a configurable point these results are written to disk to improve resource utilization.
Describe the feature, and optionally a solution or implementation and any alternatives
Ingestor.ingest(spill_threshold=Union[None, int], spill_path=str, return_type=Union[Path, Dict]...)
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Python Ingestor.ingest implementation and the ingest method's result accumulation. Trace how completed job results are retained, then define the configurable spill threshold, disk path, and return types; done means large runs no longer keep all results in memory and the requested Dict or Path result is returned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100