aws / aws/amazon-q-developer-cli

Feature request: proactive pruning of context

Open
#2,211 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2k
Forks
439
PR merge metrics
No merged PRs in 30d

Description

My typical Q sessions look something like:
1. Prompt Q to implement a change. I'll go back and forth with Q a bit on the details.
2. When it looks about right, tell Q to run test, lint, and build commands and debug errors until the build passes.

During step 2, my context frequently fills up for a few reasons:
1. Build logs are often verbose. Most of the logs are worthless, and usually just the last few lines have the error message worth looking at.
2. Linters, type checkers, and unit tests often return detailed error messages for issues.
3. When an auto-formatter runs on a file, Q needs to reread the file. Re-reads duplicate the file contents in the context.

As the context fills up, Q's speed drops, its accuracy decreases, and eventually I hit an auto-compact.

A theme for these context fillers is that they are important for only a short period of time. After a unit test is fixed for example, I'd like the context related to it to be cleared out. Or once a formatter runs, I'd like to remove any old versions of a file.

There are a couple current solutions but they fall short:
1. Run /compact. Does not work well in my experience. Too much of the useful context in the initial prompts disappears and Q needs to basically refill its context with the important files.
2. Have Q delegate some of these test/build/lint tasks to another Q process which can fix them and return a short summary. Works better in my experience, but a bit tricky to get working correctly.

Proposal: It'd be great if Q had a proactive context management process/agent which runs outside of the chat loop and prunes the context over time. It could remove context that's wholly irrelevant (e.g. file contents before a formatter ran) and summarize context where the detail doesn't matter (e.g. shorten a lengthy build log to "Build failed with a `KeyError` because of a typo on line 100 in my_code.py").
- Besides avoiding context overflows, it could keep the context focused on important info and prevent the LLM model from getting distracted from details that are no longer relevant.
- Since summarization is a simpler task, Q could use a faster and cheaper model for context management resulting in a performance increase and cost savings

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.