[BUG] output_json fallback double-encodes valid text-only LLM JSON
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Description
Task(output_json=Schema) can fail with a TaskOutput.json_dict validation error after a text-only LLM has returned valid JSON during the output-conversion step. Converter.to_json() and ato_json() wrap that JSON string in json.dumps(), so the single json.loads() in Task._unpack_model_output() produces another string instead of a dictionary.
Steps to Reproduce
- Use a
BaseLLMimplementation withsupports_function_calling() == False. - Execute a task configured with
output_json=Personwhose initial final answer is prose. - Return the valid JSON string
{"name": "Alice", "age": 30}when the output converter requests structured output. - The task raises a validation error while building
TaskOutput, even though the conversion response is valid JSON matching the requested model.
Both Task.execute_sync() and Task.aexecute_sync() reproduce this. Returning JSON in the initial final answer succeeds, as does using output_pydantic=Person with the same two responses.
Expected behavior
The converted task result has json_dict == {"name": "Alice", "age": 30}.
Screenshots/Code snippets
The following smaller reproduction isolates the same converter behavior without any model credentials or network access:
import json
from typing import Any
from crewai import BaseLLM
from crewai.utilities.converter import Converter
from pydantic import BaseModel
class Person(BaseModel):
name: str
age: int
class TextOnlyLLM(BaseLLM):
def call(self, messages: Any, **kwargs: Any) -> str:
return '{"name": "Alice", "age": 30}'
async def acall(self, messages: Any, **kwargs: Any) -> str:
return self.call(messages, **kwargs)
def supports_function_calling(self) -> bool:
return False
converter = Converter(
llm=TextOnlyLLM(model="offline-scripted"),
text="Alice is 30 years old.",
model=Person,
instructions="Return the person's name and age as JSON.",
)
result = json.loads(converter.to_json())
assert result == {"name": "Alice", "age": 30}, (type(result), result)
Operating System
Windows, with the project installed in an isolated venv.
Python Version
3.12.13
crewAI Version
1.15.21 from source at 894898f84c4ac0a89f24bf7bee6c381eb0e67f51.
crewAI Tools Version
Not used in this reproduction.
Virtual Environment
Venv, using the repository's frozen uv workspace dependencies.
Evidence
The public Task reproduction reports the following error in both sync and async execution:
1 validation error for TaskOutput
json_dict
Input should be a valid dictionary [type=dict_type,
input_value='{"name": "Alice", "age": 30}', input_type=str]
The runtime reproduction matrix was 2 failing fallback cases and 4 passing controls (direct JSON and output_pydantic, each sync and async). It uses a deterministic local BaseLLM subclass and real Task, Agent, Converter and Pydantic code; no live-provider failure is claimed.
Possible Solution
Preserve one JSON encoding layer in the non-function-calling output converter, with regression coverage for both execution modes and existing retry behavior. A focused fix and tests are being prepared for this issue.
Additional context
Related historical work: closed PR #2283 proposed a broader custom OpenAI output fallback and included a change in this area. It was not merged. This report supplies a current source-level and public Task reproduction of the still-present behavior; it does not claim that the general problem has never been discussed.
This issue was prepared with AI assistance and should carry the required llm-generated label. If the author cannot apply repository labels, please apply that label before review.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Converter.to_json() and ato_json(), then trace Task._unpack_model_output() for the text-only BaseLLM path. Add regression coverage for Task.execute_sync() and Task.aexecute_sync(), including existing retry behavior, and verify the converted result's json_dict is a dictionary matching Person.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100