NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: Have store task use semantic chunk image naming
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
Improvement
How would you describe the priority of this feature request
Significant improvement
Please provide a clear description of problem this feature solves
Enable the store_task to generate human meaningful filenames
Describe the feature, and optionally a solution or implementation and any alternatives
For the multimodal_test.pdf, using the store task like so:
ingestor = (
ingestor.files(documents)
.extract()
.embed()
.store()
.vdb_upload()
)
produces hash filenames:
ls stored_images/
17e0a604fbce20ab0ceeb4812370964696abd7f7.png 669fe72c7118e301a11f196a56ab3a51a29bf718.jpeg
It'd be nice if instead of hashed filenames, they followed a pattern like:
$SOURCE_FILENAME_$PAGE_NUMBER_$CHUNK_TYPE_$CHUNK_NUM.png
That way the chunks can be directly translated back to their origin content
Additional context
As a workaround, Codex wrote a gist
python image_chunk_mapping.py
Or point it at a specific DB/table:
python image_chunk_mapping.py --lancedb-uri lancedb --table-name nemo-retriever
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Python store_task implementation and tracing where extracted image chunks are written to stored_images. Use multimodal_test.pdf and the provided pipeline as the reproduction case. Done means generated filenames include the source filename, page number, chunk type, and chunk number instead of hashes, while preserving the image output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100