deepset-ai / deepset-ai/haystack
feat: support token-based budget in LostInTheMiddleRanker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
I was reading through LostInTheMiddleRanker and noticed the ranker's word_count_threshold parameter is implemented with content.split() — so it counts whitespace-separated words — but I'd guess what most users actually care about is the downstream LLM's context budget, which is measured in tokens.
The two are very different in practice. English prose is roughly 1.3 tokens per word, but code-heavy documents can be 2.5–4 tokens per word (every (, ., = becomes its own token), non-Latin scripts are similarly inflated, and things like URLs or hashes can be 5–15 tokens for a single "word". So if I set word_count_threshold=8000 thinking I'm targeting an 8k context window, I might actually be sending 20k+ tokens downstream — and either hit a hard API error or, worse, get a silently-truncated prompt where the LLM answers without context I thought I'd given it. It's a quiet failure mode that's hard to notice until something goes wrong.
Describe the solution you'd like
A small, additive change: a new keyword-only count_mode: Literal["word", "char", "token"] = "word" parameter on LostInTheMiddleRanker, plus an optional tokenizer_encoding string for when token mode is selected. Default stays "word" so existing users see no behavior change at all.
The nice thing is that there's already a precedent for exactly this pattern in the codebase - RecursiveDocumentSplitter uses the same Literal["word", "char", "token"] signature, lazy-imports tiktoken, defaults to the o200k_base encoding, and initializes the tokenizer in warm_up(). So this proposal would just be applying an already-accepted pattern to a sister component that needs it for the same reason: fitting content inside an LLM's real budget.
Describe alternatives you've considered
A few things I thought about and rejected:
- Rename word_count_threshold -> threshold. Cleaner, but breaks every existing user and every serialized pipeline using this ranker. Not worth it. Keeping the name (with a docstring note about the slight inaccuracy when in token mode) seems much less disruptive. An alias could be a possible follow-up, but I'd keep it out of this PR.
- Pre-truncate documents myself before the ranker. Defeats the purpose — the ranker exists precisely so I don't have to. Also, pre-truncation doesn't compose with the U-shaped reordering, since the reordering itself decides which docs end up included.
- Just document the limitation. A docstring warning helps a careful reader, but it doesn't fix the silent-failure case where a user has set a "reasonable" word threshold and is getting truncated prompts they don't know about.
Additional context
The existing precedent I mentioned is in haystack/components/preprocessors/recursive_splitter.py (the split_unit parameter, the tiktoken_imports lazy import, and the warm_up() block that calls tiktoken.get_encoding("o200k_base")). If a token-counting mode is the right call here, I'd want to mirror those exact choices for consistency rather than introduce new ones.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate LostInTheMiddleRanker and read how word_count_threshold currently uses content.split(). Compare its counting options, lazy tiktoken import, and warm_up() behavior with haystack/components/preprocessors/recursive_splitter.py. Done means word, char, and token modes are supported with the stated defaults while existing word-mode behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100