Design new LLM database schema
Open
Nobody has claimed this yet.
design
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 998
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 15
Description
The LLM database schema for storing responses needs to evolve to support some new features:
- "Mock" responses. Sometimes it's useful to run a prompt that contains several previous messages which didn't actually come from a model - for example, prompting Claude 3 Haiku with a bunch of examples that are input as previous user messages. Differentiating these from actual text returned by a model is important to support LLM's goal of recording responses from models over time so you can analyze them later.
- Another Claude feature: prompt caching. This is a larger feature to decide how to build, but since one of the uses of prompt caching is for longer conversations it's important to be able to remember at what point in the conversation the
"cache_control": {"type": "ephemeral"}rule was injected. - Images! For prompts that use images one of the following three needs to be stored: the image URL (for images that were referenced by their URL) the image file path (for images that were a file on disk) or the raw bytes of the image itself (for cases where we want that stored in the database as opposed to living separately on disk)
- Other multi-modal data: Gemini Pro accepts PDF and audio and video, fer example
- That Claude trick where you can pre-fill the prefix for the response, passing
<!doctype html><html>to ensure an HTML response for example. If that trick is used it should be recorded in the database as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping the existing LLM response schema and its storage or migration entry points, then trace how prompts, responses, and multimodal content are represented. The design should cover mock responses, prompt-cache markers, images, other media, and prefilled response prefixes, with tests or migration checks demonstrating that each can be stored and recovered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100