simonw / simonw/llm

Possible schema issue with enum output

Open
#815 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12.5k
Forks
998
Avg merge
6d 12h
Merged PRs (30d)
15

Description

I'm working with the new schema support (I just added it for llm-ollama, for context), and I am running into something I don't quite understand which may be a bug. I am using the llm-ollama plugin for all these examples.

Take this brief demo snippet:

#!/usr/bin/env python

import llm

CATEGORY_SCHEMA = {
    "type": "string",
    "pattern": "^[a-zA-Z0-9]*$",
}

model = llm.get_model("llama3.2")
response = model.prompt(
    "Is 11:30 AM in the night or the day?",
    schema=CATEGORY_SCHEMA
)
print(f"Original response: '{response.text()}'")
stripped = response.text().replace('"', '')
print(f"Stripped quotes: '{stripped}'")
print("Ordinals:")
for x in response.text():
    print(f"'{x}': {ord(x)}")

When I run it, I sometimes get an output like:

$ ~/tmp/schema-demo.py
Original response: '"Night" '
Stripped quotes: 'Night '
Ordinals:
'"': 34
'N': 78
'i': 105
'g': 103
'h': 104
't': 116
'"': 34
' ': 32

The schema is functioning, since if I exclude it I usually get a sentence-long response. However, the schema-constrained response contains quote and space characters which are disallowed by the pattern.

I am still learning about LLMs, so if this is just an expected sort of weirdness I'll accept it and add some additional validation on my end. I just wanted to report it in case it represented a real issue that was worth addressing. Thanks for your hard work on this library, it's been a joy to work with!

Fun note: if I use the LLM command-r7b, I get even more bonkers output:

$ ~/tmp/schema-demo.py
Original response: '"Night"
                                          '
Stripped quotes: 'Night
                                          '
Ordinals:
'"': 34
'N': 78
'i': 105
'g': 103
'h': 104
't': 116
'"': 34
'
': 10
'       ': 9
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
'       ': 9
' ': 32
' ': 32
' ': 32
' ': 32
'       ': 9
'       ': 9
' ': 32
' ': 32

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the schema-demo.py example with the llm-ollama plugin, the llama3.2 model, and command-r7b, comparing responses with and without the schema. Start by tracing the schema-support behavior involved in model.prompt(); done means establishing whether the quoted or whitespace-padded output violates the requested pattern and documenting or addressing the behavior accordingly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.