[BUG] response_model is ignored by acall() when streaming is enabled
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Description
await llm.acall(messages, response_model=SomeModel) returns the model's raw text instead
of the structured output when streaming is enabled. _ahandle_streaming_response accepts
response_model, documents it, and never uses it.
LLM.acall forwards the argument correctly:
if self._effective_stream():
return await self._ahandle_streaming_response(
params=params,
callbacks=callbacks,
available_functions=available_functions,
from_task=from_task,
from_agent=from_agent,
response_model=response_model, # <-- arrives here
)
The handler declares the parameter at
llm.py:1533
and documents it as "Optional response model", but the name never appears again in the
function body. The success path returns full_response — the accumulated stream text —
verbatim:
The other three handlers all consume it and convert via InternalInstructor:
| handler | line | consumes response_model? |
|---|---|---|
_handle_streaming_response |
llm.py:1040 |
yes |
_handle_non_streaming_response |
llm.py:1242 |
yes |
_ahandle_non_streaming_response |
llm.py:1397 |
yes |
_ahandle_streaming_response |
— | no |
Reproduction
Replace litellm.completion/acompletion with a stub that streams a prose answer (not
JSON), replace InternalInstructor so no second LLM call is needed, then call all four
paths with the same response_model:
class Landmark(BaseModel):
city: str
population: int
# What a model streams when it answers in prose. Deliberately NOT valid JSON.
PROSE = "The Eiffel Tower is in Paris, which has about 2,100,000 residents."
llm = LLM(model="gpt-4o", is_litellm=True, api_key="test", stream=stream)
out = llm.call(msgs, response_model=Landmark) # or: await llm.acall(...)
Result on main (ceed4a3):
sync stream=False -> structured | {"city":"Paris","population":2100000}
sync stream=True -> structured | {"city":"Paris","population":2100000}
async stream=False -> structured | {"city":"Paris","population":2100000}
async stream=True -> RAW TEXT | The Eiffel Tower is in Paris, which has about 2,100,000 residents.
Three of four paths honour the contract. The fourth returns prose.
Why this matters
response_model is the structured-output contract, so callers act on the return value
without re-checking it:
- Silent corruption, not a loud failure. The return type is
strin both cases, so
nothing raises at the boundary. Downstreammodel_validate_json/json.loadseither
throws far from the cause, or — worse — a prose answer that happens to contain
JSON-looking text parses into the wrong object. - Reachable through normal configuration. No malformed input and no unusual setup:
stream=Trueplusresponse_modelplusacallis the ordinary combination for a
streaming async agent that wants typed output. Both are documented, supported options. - Streaming is exactly where async is used. The path that silently drops the parameter
is the one an async streaming application takes by default. - The sync/async asymmetry makes it a trap. Code developed and tested synchronously
behaves correctly, then quietly stops honouringresponse_modelwhen moved toacall
with no signature change and no warning.
Expected behaviour
await llm.acall(msgs, response_model=M) with stream=True returns the same structured
JSON string as the other three paths.
Fix
Mirror the block that _handle_streaming_response already has, in the no-tool-call return
path of _ahandle_streaming_response:
if response_model and self.is_litellm:
instructor_instance = InternalInstructor(
content=full_response,
model=response_model,
llm=self,
)
result = instructor_instance.to_pydantic()
structured_response = result.model_dump_json()
self._handle_emit_call_events(
response=structured_response,
...
)
return structured_response
The self.is_litellm guard and the InternalInstructor call match the sync handler
exactly, so behaviour is consistent across all four paths and native providers are
unaffected.
I have a PR ready with this change plus five tests (one regression, four controls covering
the three already-correct paths and the no-response_model case).
Environment
- crewAI
main@ceed4a3ff71b5b4cb0ca316b4178ffcce74a53b2 - Python 3.12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/crewai/src/crewai/llm.py at _ahandle_streaming_response and compare its no-tool-call return path with _handle_streaming_response, _handle_non_streaming_response, and _ahandle_non_streaming_response. Use the reproduction with a response_model and streaming async call; done means the async streaming path returns structured JSON consistently with the other three paths and the five described tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100