danielmiessler / danielmiessler/Fabric

[Feature request]: Handle Splitting?

Open
#1,055 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
43.9k
Forks
4.3k
Avg merge
1d 4h
Merged PRs (30d)
8

Description

### What do you need?

It would be great if fabric could automatically handle splitting/chunking for text that is too large for a given model.

From what I understand, this would need:
- Information about the token limit of the model being used
- A way to count the tokens in a specific request
- A range of splitting options (character, word, sentence, recursive, semantic etc.) to choose from
- Possibly a way to select the LLM used for semantic splitting

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating how model limits and request sizing are handled, then assess the proposed character, word, sentence, recursive, and semantic splitting options. Done would require an agreed design and implementation covering the supported splitting behavior and any semantic-model selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.