posit-dev / posit-dev/connect-extensions

Chat with content: hitting TPM limits very fast, even with quite basic static content (R markdown) - can chunking be used

Open
#295 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
10
Forks
8
Avg merge
5d 18h
Merged PRs (30d)
1

Description

Dear,

I have deployed "chat with content" and I am trying to use it on (quite minimal) R markdown reports. I have set "chat with content" up to use OpenAI - I tried multiple iterations with gpt-4.1-mini, gpt-5-mini, ... but I am always receiving an error: Request too large for gpt-4.1-long-context in organization <my-org> on tokens per min (TPM): Limit 500000, Requested <number larger than 500k>. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}.

I am a bit surprised about the number of tokens that even relatively basic R markdown reports take up, but I am wondering:

  • Why, regardless of what I specify under CHATLAS_CHAT_ARGS, do I always see gpt-4.1-long-context in the error message?
  • Is there a way to implement some kind of RAG "chunking" to avoid exceeding rate limits?

Thanks in advence,

FM Kerckhof

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how CHATLAS_CHAT_ARGS selects the OpenAI model in the chat-with-content extension, then reproduce the TPM error with a minimal R Markdown report. Check the existing content retrieval path for any chunking behavior. Done should include a clear explanation of the model selection and a defined approach for handling reports that exceed TPM limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown, r
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.