NVIDIA / NVIDIA/NeMo-Retriever

Unify normalized text chunking for HTML and plain-text inputs

Open
#2,320 0 comments 0 reactions 1 assignee View on GitHub

@jioffe502 is already working on this.

Since Jul 14, 2026.

Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Context

HTML ingestion converts source content to an in-memory Markdown string with MarkItDown, then builds token chunks and ingestion rows in _markdown_to_chunks_df(). Native .txt, .md, .json, and .sh inputs perform nearly identical chunking and row construction in the text module.

Current behavior is correct, but the duplicated implementation could drift when tokenizer settings, chunk metadata, or empty-input handling change.

Goal

Introduce one shared in-process text-to-chunks implementation at the seam after format normalization:

HTML -> MarkItDown --+
                     +-> shared text-to-chunks module
Text/Markdown -------+

Acceptance criteria

  • HTML and native text adapters call one shared text-to-chunks implementation.
  • Existing text, metadata, source paths, and chunk boundaries remain unchanged.
  • Representative HTML, Markdown, TXT, JSON, and shell inputs retain equivalent behavior.
  • Tests verify behavior through the existing ingestion interface.
  • Redundant implementation-level tests are removed where equivalent public behavior coverage exists.

Non-goals

  • New Markdown, JSON, or shell parsers.
  • A new chunking strategy.
  • Retrieval-quality or performance changes.

This is low-priority maintainability work and should not block extension-support fixes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.