michaelfeil / michaelfeil/infinity
Add a TextSplitter in LangChain to share the model of the embedding model
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### Feature request
Have you ever though to add an API endpoint that can serve as well as TextSplitter ?
It would replace the need to load in memory the same model for the text Chunker and the Embedder
https://python.langchain.com/docs/modules/data_connection/document_transformers/split_by_token#sentencetransformers
### Motivation
Create a LangChain TextSplitter that is based on the model Tokenizer to chunk long documents
### Your contribution
I feel ignorant about the AI domain (lack of knowledge)
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by locating the existing embedding model, tokenizer, and serving APIs, then compare them with LangChain's SentenceTransformers-based splitter. Done means a tokenizer-backed TextSplitter can share the loaded model with the embedder, with tests covering chunking and model reuse.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100