spacedriveapp / spacedriveapp/spacebot
Prompt cache invalidated with each channel query
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 2.4k
- Forks
- 367
- PR merge metrics
- No merged PRs in 30d
Description
I started testing Spacebot and noticed that the prompt cache was never being used when chatting with the Channel agent.
I'm using local models via llama.cpp/llama-proxy/litellm. I have an fresh spacebot instance and the inference server was not receiving any other queries that could interfere with prompt caching.
When querying the Cortex agent directly via the web-ui, prompt caching does work as expected.
I did a quick diff check on the Channel agent messages and it revealed that the system prompt changes with every request.
The offending section appears to be related to the current system time:
This seems to be by design:
https://github.com/spacedriveapp/spacebot/blob/6f81f3979e1b5592e9909f66b53eaad593196a41/src/agent/status.rs#L122
but perhaps there is another way to accomplish this which is not so expensive? I'm not entirely sure if having the agent call a tool to check the current timestamp when relevant in a conversation would work reliably (either based on just knowing about the tool, or tweaking the system prompt to nudge it to do so eagerly)
(Side-note, is that newline supposed to be there between ## System and Time:?)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/agent/status.rs around line 122 and compare the Channel agent's generated system prompt with the Cortex agent's prompt through the web UI. Reproduce the changing current-time section and verify how it affects prompt-cache reuse. Done means the Channel prompt no longer invalidates the cache unnecessarily while still providing the required time information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100