Possible schema issue with enum output
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 998
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 15
Description
I'm working with the new schema support (I just added it for llm-ollama, for context), and I am running into something I don't quite understand which may be a bug. I am using the llm-ollama plugin for all these examples.
Take this brief demo snippet:
#!/usr/bin/env python
import llm
CATEGORY_SCHEMA = {
"type": "string",
"pattern": "^[a-zA-Z0-9]*$",
}
model = llm.get_model("llama3.2")
response = model.prompt(
"Is 11:30 AM in the night or the day?",
schema=CATEGORY_SCHEMA
)
print(f"Original response: '{response.text()}'")
stripped = response.text().replace('"', '')
print(f"Stripped quotes: '{stripped}'")
print("Ordinals:")
for x in response.text():
print(f"'{x}': {ord(x)}")
When I run it, I sometimes get an output like:
$ ~/tmp/schema-demo.py
Original response: '"Night" '
Stripped quotes: 'Night '
Ordinals:
'"': 34
'N': 78
'i': 105
'g': 103
'h': 104
't': 116
'"': 34
' ': 32
The schema is functioning, since if I exclude it I usually get a sentence-long response. However, the schema-constrained response contains quote and space characters which are disallowed by the pattern.
I am still learning about LLMs, so if this is just an expected sort of weirdness I'll accept it and add some additional validation on my end. I just wanted to report it in case it represented a real issue that was worth addressing. Thanks for your hard work on this library, it's been a joy to work with!
Fun note: if I use the LLM command-r7b, I get even more bonkers output:
$ ~/tmp/schema-demo.py
Original response: '"Night"
'
Stripped quotes: 'Night
'
Ordinals:
'"': 34
'N': 78
'i': 105
'g': 103
'h': 104
't': 116
'"': 34
'
': 10
' ': 9
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 32
' ': 9
' ': 32
' ': 32
' ': 32
' ': 32
' ': 9
' ': 9
' ': 32
' ': 32
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the schema-demo.py example with the llm-ollama plugin, the llama3.2 model, and command-r7b, comparing responses with and without the schema. Start by tracing the schema-support behavior involved in model.prompt(); done means establishing whether the quoted or whitespace-padded output violates the requested pattern and documenting or addressing the behavior accordingly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100