NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: Ingest pdf with custom metadata
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Currently preventing usage
Please provide a clear description of problem this feature solves
Is it possible to take custom metadata along when sending the data to nv-ingest?
Example below ingest a pdf with native request. How to add custom metadata with it so in the vector db it has those fields too.
` data = {
"collection_name": collection_name,
"extraction_options": {
"extract_text": True,
"extract_tables": True,
"extract_charts": True,
"extract_images": False, # Set to True if you want to extract images, ensure the VLM model is deployed
"extract_method": "pdfium",
"text_depth": "page",
},
"split_options": {
"chunk_size": 1024,
"chunk_overlap": 150
}
}
form_data = aiohttp.FormData()
for file_path in files[1:]:
print(f"Uploading {file_path} to {collection_name}")
form_data.add_field("documents", open(file_path, "rb"), filename=os.path.basename(file_path), content_type="application/pdf")
form_data.add_field("data", json.dumps(data), content_type="application/json")
async with aiohttp.ClientSession() as session:
try:
async with session.post(f"{BASE_URL}/v1/documents", data=form_data) as response: # Replace with session.patch for reingesting
await print_response(response)
except aiohttp.ClientError as e:
print(f"Error: {e}")`
Describe the feature, and optionally a solution or implementation and any alternatives
A new feature example
data = { "collection_name": collection_name, "extraction_options": { "extract_text": True, "extract_tables": True, "extract_charts": True, "extract_images": False, # Set to True if you want to extract images, ensure the VLM model is deployed "extract_method": "pdfium", "text_depth": "page", }, "split_options": { "chunk_size": 1024, "chunk_overlap": 150 }, // add custom metadata "custom_metadata": {"abc":123} }
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue identifies the /v1/documents endpoint and a proposed custom_metadata field, but no repository files or tests. Locate the endpoint implementation and trace how document metadata is passed through ingestion; verify that supplied fields reach the vector database and add or run the relevant API tests if present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100