deeppavlov / deeppavlov/AutoIntent
`LLMDescriptionScorer` prompt puts the utterance before the descriptions, defeating provider prefix caching
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
What happens
LLMDescriptionScorer._create_prompt (llm_encoder.py:155-200) lays the single user message out as
- two-line header,
<text_sample>{utterance}</text_sample>,<possible_intent_descriptions>— all N descriptions, identical for every utterance of a run and the bulk of the tokens (77–150 descriptions on banking77/clinc150),<instructions>.
Provider prompt/prefix caching (OpenAI automatic prompt caching, Anthropic cache, OpenRouter pass-through) only matches the longest identical prefix, and here the prefix diverges at the utterance, so the descriptions block is re-billed at full input price on every call. Input tokens are essentially the whole cost of each call (the output is a short JSON of indices).
Proposed
Order the message header → descriptions → instructions → utterance (utterance last). The shared prefix then covers everything but the utterance, and cached input tokens are billed at the provider's discounted rate across different utterances, not just exact repeats. Darinochka/AutoIntent-experiments#43 saw the exact-repeat effect already: a byte-identical dry-run repeat cost $0.0054/call vs $0.0308 cold on OpenRouter.
Caveats: the prompt text is part of the StructuredOutputCache key, so existing caches miss once; and the change moves the question relative to the context, so quality should be re-checked on one dataset before switching the default.
Follow-up from #350.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/autointent/modules/scoring/_description/llm_encoder.py, especially LLMDescriptionScorer._create_prompt at lines 155–200, and compare the current prompt ordering with the proposed shared-prefix layout. Verify the StructuredOutputCache impact, then evaluate the changed default on one dataset to confirm intent-classification quality remains acceptable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100