simonw / simonw/llm

How to handle fake messages that were not part of real coversations?

Open
#468 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design
Dominant language
Python
Stars
12.5k
Forks
998
Avg merge
3d 13h
Merged PRs (30d)
10

Description

For Claude models a common pattern is to provide examples as a bunch of fake user/assistant messages:

curl https://api.anthropic.com/v1/messages \
     --header "x-api-key: $ANTHROPIC_API_KEY" \
     --header "anthropic-version: 2023-06-01" \
     --header "content-type: application/json" \
     --data \
'{
    "model": "claude-3-haiku-20240307",
    "max_tokens": 1024,
    "system": "You classify news stories as tech, opinion, world or usa",
    "messages": [
        {"role": "user", "content": "New Apple trackpad breeds nasty fungus"},
        {"role": "assistant", "content": "tech"},
        {"role": "user", "content": "I do not like how the world is going"},
        {"role": "assistant", "content": "opinion"},
        {"role": "user", "content": "Potential war brewing in Spain"},
        {"role": "assistant", "content": "world"},
        {"role": "user", "content": "Who will be the next senator from Vermont?"},
        {"role": "assistant", "content": "usa"},
        {"role": "user", "content": "Microsoft release Windows 334"}
    ]
}'

llm doesn't really support this yet - the conversation mechanism is just for when you reply to threads, but there isn't a clean way to feed it a whole bunch of "fake" conversation like this as part of a prompt.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests; start by reviewing the existing conversation mechanism and how the CLI accepts prompt messages. Determine how Claude-style sequences of fake user and assistant messages should be represented, and consider the example classification request the issue provides as the completion target.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.