microsoft / microsoft/azurechat
Bing Search extension reaches the maximum of the token
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 1.3k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 1
Description
Hi. First of all, this extension works fine on GPT- 4 model. But the performance on GPT-4 is very slow at the moment. So I switched to using GPT 3.5 1106 instead but I got this error message for a simple question "give me azure latest news"
"This model's maximum context length is 16187 tokens, however you requested 16386 tokens (16038 in your prompt; 348 for the completion). Please reduce your prompt; or completion length."
I am not quite sure why it uses more 16K token for a single search. But it could probably take the entire Bing result and pass to GPT3.5 model.
How can we limit the number of results we want from Bing API and also in the Extension setting, It would be nice if we could specify to include or exclude fields that we don't need to pass to the model, it will reduce a lot of token consumption.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Bing Search extension and the Bing API response path described in the report, then locate where search results are passed to the GPT-3.5 model and where extension settings are defined. Done means search result size or included fields can be limited so a simple query stays within the model context limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100