spacedriveapp / spacedriveapp/spacebot

Prompt cache invalidated with each channel query

Open
#481 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.4k
Forks
367
PR merge metrics
No merged PRs in 30d

Description

I started testing Spacebot and noticed that the prompt cache was never being used when chatting with the Channel agent.

I'm using local models via llama.cpp/llama-proxy/litellm. I have an fresh spacebot instance and the inference server was not receiving any other queries that could interfere with prompt caching.

When querying the Cortex agent directly via the web-ui, prompt caching does work as expected.

I did a quick diff check on the Channel agent messages and it revealed that the system prompt changes with every request.

The offending section appears to be related to the current system time:

Image

This seems to be by design:
https://github.com/spacedriveapp/spacebot/blob/6f81f3979e1b5592e9909f66b53eaad593196a41/src/agent/status.rs#L122
but perhaps there is another way to accomplish this which is not so expensive? I'm not entirely sure if having the agent call a tool to check the current timestamp when relevant in a conversation would work reliably (either based on just knowing about the tool, or tweaking the system prompt to nudge it to do so eagerly)

(Side-note, is that newline supposed to be there between ## System and Time:?)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/agent/status.rs around line 122 and compare the Channel agent's generated system prompt with the Cortex agent's prompt through the web UI. Reproduce the changing current-time section and verify how it affects prompt-cache reuse. Done means the Channel prompt no longer invalidates the cache unnecessarily while still providing the required time information.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.