NVIDIA / NVIDIA/NeMo-Retriever

[FEA]: Ingest pdf with custom metadata

Open
#647 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Currently preventing usage

Please provide a clear description of problem this feature solves

Is it possible to take custom metadata along when sending the data to nv-ingest?
Example below ingest a pdf with native request. How to add custom metadata with it so in the vector db it has those fields too.

` data = {
"collection_name": collection_name,
"extraction_options": {
"extract_text": True,
"extract_tables": True,
"extract_charts": True,
"extract_images": False, # Set to True if you want to extract images, ensure the VLM model is deployed
"extract_method": "pdfium",
"text_depth": "page",
},
"split_options": {
"chunk_size": 1024,
"chunk_overlap": 150
}
}

form_data = aiohttp.FormData()
for file_path in files[1:]:
    print(f"Uploading {file_path} to {collection_name}")
    form_data.add_field("documents", open(file_path, "rb"), filename=os.path.basename(file_path), content_type="application/pdf")

form_data.add_field("data", json.dumps(data), content_type="application/json")

async with aiohttp.ClientSession() as session:
    try:
        async with session.post(f"{BASE_URL}/v1/documents", data=form_data) as response: # Replace with session.patch for reingesting
            await print_response(response)
    except aiohttp.ClientError as e:
        print(f"Error: {e}")`
Describe the feature, and optionally a solution or implementation and any alternatives

A new feature example
data = { "collection_name": collection_name, "extraction_options": { "extract_text": True, "extract_tables": True, "extract_charts": True, "extract_images": False, # Set to True if you want to extract images, ensure the VLM model is deployed "extract_method": "pdfium", "text_depth": "page", }, "split_options": { "chunk_size": 1024, "chunk_overlap": 150 }, // add custom metadata "custom_metadata": {"abc":123} }

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue identifies the /v1/documents endpoint and a proposed custom_metadata field, but no repository files or tests. Locate the endpoint implementation and trace how document metadata is passed through ingestion; verify that supplied fields reach the vector database and add or run the relevant API tests if present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.