michaelfeil / michaelfeil/infinity

Add a TextSplitter in LangChain to share the model of the embedding model

Open
#193 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### Feature request

Have you ever though to add an API endpoint that can serve as well as TextSplitter ?
It would replace the need to load in memory the same model for the text Chunker and the Embedder

https://python.langchain.com/docs/modules/data_connection/document_transformers/split_by_token#sentencetransformers

### Motivation

Create a LangChain TextSplitter that is based on the model Tokenizer to chunk long documents

### Your contribution

I feel ignorant about the AI domain (lack of knowledge)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by locating the existing embedding model, tokenizer, and serving APIs, then compare them with LangChain's SentenceTransformers-based splitter. Done means a tokenizer-backed TextSplitter can share the loaded model with the embedder, with tests covering chunking and model reuse.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.