posit-dev / posit-dev/positron

Assistant: Send console history entries most relevant to query, rather than most recent

Open
#8,914 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: assistant
Dominant language
TypeScript
Stars
4.3k
Forks
184
Avg merge
1d 9h
Merged PRs (30d)
195

Description

(via a tech demo by @georgestagg)

Currently, Positron Assistant sends the most recent console history entries to the model as context. Entries are prioritized based only on their recency, so we just start with the newest entries and keep going until we run out of budget.

https://github.com/posit-dev/positron/blob/36265f9fd9c9ecebc6208acb242c2ca56f0c0583/src/vs/workbench/contrib/chat/browser/contrib/chatRuntimeSessionContext.ts#L346-L360

It is true that the most recent entries are overall more likely to be relevant than older entries, but it shouldn't be the only factor. In particular, the user may be asking about something other than their most recent query, and it is likely that code run further in the history is important as context to the question.

To improve Assistant's ability to use console history as context without stuffing the whole history into the context, we should consider using something like BM25 to help select the most relevant console entries.

(We could also use something like this when selecting variables, if there are a large number of them.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/vs/workbench/contrib/chat/browser/contrib/chatRuntimeSessionContext.ts#L346-L360, where recent console history entries are selected for Assistant context. Review how the current recency-based selection works and investigate BM25 or a similar relevance-ranking approach. Done should mean relevant console entries are selected within the context budget, with behavior validated by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.