NVIDIA / NVIDIA/NeMo-Retriever

[FEA]: Have store task use semantic chunk image naming

Open
#2,618 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Significant improvement

Please provide a clear description of problem this feature solves

Enable the store_task to generate human meaningful filenames

Describe the feature, and optionally a solution or implementation and any alternatives

For the multimodal_test.pdf, using the store task like so:

ingestor = ( 
  ingestor.files(documents)
  .extract()
  .embed()
  .store()
  .vdb_upload()
)

produces hash filenames:

ls stored_images/
17e0a604fbce20ab0ceeb4812370964696abd7f7.png  669fe72c7118e301a11f196a56ab3a51a29bf718.jpeg

It'd be nice if instead of hashed filenames, they followed a pattern like:
$SOURCE_FILENAME_$PAGE_NUMBER_$CHUNK_TYPE_$CHUNK_NUM.png

That way the chunks can be directly translated back to their origin content

Additional context

As a workaround, Codex wrote a gist

python image_chunk_mapping.py

Or point it at a specific DB/table:

python image_chunk_mapping.py --lancedb-uri lancedb --table-name nemo-retriever

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Python store_task implementation and tracing where extracted image chunks are written to stored_images. Use multimodal_test.pdf and the provided pipeline as the reproduction case. Done means generated filenames include the source filename, page number, chunk type, and chunk number instead of hashes, while preserving the image output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.