MemberJunction / MemberJunction/MJ

Revamp Chunking with Chunking Base Class and Implementation Subclasses

Open
#518 0 comments 0 reactions 1 assignee Claimed by @nico-oz View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

Currently the Content Autotagging project is restricted to just using fixed length chunking based on the maximum input token limit for the selected model. The idea would be to offer several chunking strategies ordered by priority and that can be chosen from at runtime. There will be a base Chunking and provider subclasses. In particular will be pursuing an implementation of a tool like [jina](https://jina.ai/reader/).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.