NVIDIA / NVIDIA/NeMo-Retriever

[EPIC] Create purely pythonic `nv_ingest_common` module

Open
#351 0 comments 0 reactions 1 assignee View on GitHub

@jdye64 is already working on this.

Since Jan 20, 2025.

epic
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

There is currently duplicative code in both nv_ingest and nv_ingest_client. The duplicative code ranges from Pydantic schemas that are needed in both modules to utility functions needed in both locations. The goal of this Epic is to setup a nv_ingest_common module that takes the duplicative code from each independent module and places it in a common/shared location.

Furthermore this module should be a lite and purely python as possible. This module should be written in a fashion where other libraries or runtimes can import it and use in a fashion similar to nv_ingest_runtime. This of course means that nv_ingest_runtime will need some refactoring to work with code that is moved to this common module.

This Epic can be broken down into the follows issues.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.