crewAIInc / crewAIInc/crewAI

[BUG] output_json fallback double-encodes valid text-only LLM JSON

Open
#7,429 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

Task(output_json=Schema) can fail with a TaskOutput.json_dict validation error after a text-only LLM has returned valid JSON during the output-conversion step. Converter.to_json() and ato_json() wrap that JSON string in json.dumps(), so the single json.loads() in Task._unpack_model_output() produces another string instead of a dictionary.

Steps to Reproduce
  1. Use a BaseLLM implementation with supports_function_calling() == False.
  2. Execute a task configured with output_json=Person whose initial final answer is prose.
  3. Return the valid JSON string {"name": "Alice", "age": 30} when the output converter requests structured output.
  4. The task raises a validation error while building TaskOutput, even though the conversion response is valid JSON matching the requested model.

Both Task.execute_sync() and Task.aexecute_sync() reproduce this. Returning JSON in the initial final answer succeeds, as does using output_pydantic=Person with the same two responses.

Expected behavior

The converted task result has json_dict == {"name": "Alice", "age": 30}.

Screenshots/Code snippets

The following smaller reproduction isolates the same converter behavior without any model credentials or network access:

import json
from typing import Any

from crewai import BaseLLM
from crewai.utilities.converter import Converter
from pydantic import BaseModel


class Person(BaseModel):
    name: str
    age: int


class TextOnlyLLM(BaseLLM):
    def call(self, messages: Any, **kwargs: Any) -> str:
        return '{"name": "Alice", "age": 30}'

    async def acall(self, messages: Any, **kwargs: Any) -> str:
        return self.call(messages, **kwargs)

    def supports_function_calling(self) -> bool:
        return False


converter = Converter(
    llm=TextOnlyLLM(model="offline-scripted"),
    text="Alice is 30 years old.",
    model=Person,
    instructions="Return the person's name and age as JSON.",
)
result = json.loads(converter.to_json())
assert result == {"name": "Alice", "age": 30}, (type(result), result)
Operating System

Windows, with the project installed in an isolated venv.

Python Version

3.12.13

crewAI Version

1.15.21 from source at 894898f84c4ac0a89f24bf7bee6c381eb0e67f51.

crewAI Tools Version

Not used in this reproduction.

Virtual Environment

Venv, using the repository's frozen uv workspace dependencies.

Evidence

The public Task reproduction reports the following error in both sync and async execution:

1 validation error for TaskOutput
json_dict
  Input should be a valid dictionary [type=dict_type,
  input_value='{"name": "Alice", "age": 30}', input_type=str]

The runtime reproduction matrix was 2 failing fallback cases and 4 passing controls (direct JSON and output_pydantic, each sync and async). It uses a deterministic local BaseLLM subclass and real Task, Agent, Converter and Pydantic code; no live-provider failure is claimed.

Possible Solution

Preserve one JSON encoding layer in the non-function-calling output converter, with regression coverage for both execution modes and existing retry behavior. A focused fix and tests are being prepared for this issue.

Additional context

Related historical work: closed PR #2283 proposed a broader custom OpenAI output fallback and included a change in this area. It was not merged. This report supplies a current source-level and public Task reproduction of the still-present behavior; it does not claim that the general problem has never been discussed.

This issue was prepared with AI assistance and should carry the required llm-generated label. If the author cannot apply repository labels, please apply that label before review.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Converter.to_json() and ato_json(), then trace Task._unpack_model_output() for the text-only BaseLLM path. Add regression coverage for Task.execute_sync() and Task.aexecute_sync(), including existing retry behavior, and verify the converted result's json_dict is a dictionary matching Person.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.