NVIDIA / NVIDIA/GenerativeAIExamples
Internal server error for role orders in LLM inference
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.2k
- Forks
- 1.1k
- Avg merge
- 10h 15m
- Merged PRs (30d)
- 1
Description
Hi,
I am getting 500s with description
chat messages must alternate roles between 'user' and 'assistant'. Message may have a leading 'system' role message
and
"Internal Server Error\",\"status\":500,\"detail\":\"Last message role should be 'user'
I think this order validation is unnecessary. Most of the models (e.g. mixtrals, llama3, gemma etc) are perfectly fine with any order of roles. This claim can be validated on groq's playground: https://console.groq.com/playground
This issue currently breaks some existing patterns, like continuations (without the user explicitly saying "continue") or in some cases running agents with observations etc.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the validation that produces the quoted role-order and last-message errors, then reproduce a request with continuation or agent-observation messages. Done means supported non-alternating role sequences no longer return HTTP 500 while the intended validation behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100