NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: Embedding and VDB upload from Jsonl file
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Currently preventing usage
Please provide a clear description of problem this feature solves
I have a jsonl file with text snippets and corresponding ids that I want to embed and upload to Milvus through the Nv-Ingest python client and retrieve with LlamaIndex. It's important that the Ids are maintained through every step of this process.
Describe the feature, and optionally a solution or implementation and any alternatives
I would like a feature that would allow me to submit a jsonl file with an id field and a text field to the NV-Ingest client, which would embed and upload the text to the VDB while keeping the associated Id.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the NV-Ingest Python client's existing file-ingestion, embedding, and vector-database upload entry points. Check how records are passed to Milvus and retrieved through LlamaIndex, then define the JSONL id and text flow. Done means a JSONL file can be submitted and each associated ID remains available through embedding, upload, and retrieval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100