MemberJunction / MemberJunction/MJ
Revamp Chunking with Chunking Base Class and Implementation Subclasses
Open
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
Currently the Content Autotagging project is restricted to just using fixed length chunking based on the maximum input token limit for the selected model. The idea would be to offer several chunking strategies ordered by priority and that can be chosen from at runtime. There will be a base Chunking and provider subclasses. In particular will be pursuing an implementation of a tool like [jina](https://jina.ai/reader/).
Contributor guide
Assessment
This issue has not been assessed yet.