posit-dev / posit-dev/connect-extensions
Chat with content: hitting TPM limits very fast, even with quite basic static content (R markdown) - can chunking be used
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 10
- Forks
- 8
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 1
Description
Dear,
I have deployed "chat with content" and I am trying to use it on (quite minimal) R markdown reports. I have set "chat with content" up to use OpenAI - I tried multiple iterations with gpt-4.1-mini, gpt-5-mini, ... but I am always receiving an error: Request too large for gpt-4.1-long-context in organization <my-org> on tokens per min (TPM): Limit 500000, Requested <number larger than 500k>. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}.
I am a bit surprised about the number of tokens that even relatively basic R markdown reports take up, but I am wondering:
- Why, regardless of what I specify under
CHATLAS_CHAT_ARGS, do I always seegpt-4.1-long-contextin the error message? - Is there a way to implement some kind of RAG "chunking" to avoid exceeding rate limits?
Thanks in advence,
FM Kerckhof
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how CHATLAS_CHAT_ARGS selects the OpenAI model in the chat-with-content extension, then reproduce the TPM error with a minimal R Markdown report. Check the existing content retrieval path for any chunking behavior. Done should include a clear explanation of the model selection and a defined approach for handling reports that exceed TPM limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown, r
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100