[BUG] LLMGuardrail reports an LLM/provider error as a failed validation, and the caller retries on it

Description

LLMGuardrail.call catches every exception and returns (False, "Error while validating the task output: ..."). In this interface False means "the output violated the guardrail", so an infrastructure problem (provider outage,
expired key, rate limit) is reported to the caller as a verdict about the agent's output. The caller can't tell the two apart.

Source, at 6491f5a, lib/crewai/src/crewai/tasks/llm_guardrail.py:110-119:

try:
result = self._validate_output(task_output)
...
if result.pydantic.valid:
return True, task_output.raw
return False, result.pydantic.feedback
except Exception as e:
return False, f"Error while validating the task output: {e!s}"

Steps to reproduce

from unittest.mock import patch
from crewai.tasks.llm_guardrail import LLMGuardrail
from crewai.tasks.task_output import TaskOutput

out = TaskOutput(description="d", agent="a", raw="the agent's answer")
g = LLMGuardrail(description="must be under 100 words", llm=None)

with patch.object(LLMGuardrail, "_validate_output",
side_effect=RuntimeError("litellm.APIConnectionError: provider unavailable")):
print("outage ->", g(out))

Output:

outage -> (False, 'Error while validating the task output: litellm.APIConnectionError: provider unavailable')

A real guardrail violation returns the same shape, (False, "too long by 40 words"). A passing output returns (True, ...), so the check itself works. The problem is only that "couldn't check" and "check failed" are the same value.

Expected behavior

An exception from the LLM call should be distinguishable from a validation result. The caller should be able to decide separately what to do when the guardrail couldn't run, rather than treating it as evidence about the output.

Actual behavior, and why it matters downstream

The caller acts on the False as if the output were bad. In agent/core.py:1865-1878:

if not guardrail_result.success:
if retry_count >= self.guardrail_max_retries:
raise ValueError(f"Agent's guardrail failed validation after {self.guardrail_max_retries} retries. ...")
executor._append_message_to_state(guardrail_result.error or "Guardrail validation failed", role="user")
output = self._execute_and_build_output(executor, inputs, response_format)
return self._process_kickoff_guardrail(...)

task.py:1258 onward has the same shape. So during a provider incident you get:

1. The raw exception text appended into the agent's conversation as a user turn. Provider error bodies sometimes echo parts of the request, so this puts whatever the provider said back into the model's context.
2. A full agent re-execution, repeated up to guardrail_max_retries (default 3, agent/core.py:318). Three extra complete agent runs during an outage, each one likely to hit the same error.
3. Finally ValueError: Agent's guardrail failed validation after 3 retries, which tells whoever is on call that validation failed when the actual cause was that the provider was down.

To be clear about what this is not: nothing bad gets through. It fails closed, and I checked that specifically before filing. It's a correctness and diagnosability problem, not a bypass.

Proposed fix

Let the exception path be its own case rather than borrowing the "invalid" one. Roughly:

class GuardrailExecutionError(Exception):
"""The guardrail could not run. Not a statement about the output."""

...
except Exception as e:
raise GuardrailExecutionError(str(e)) from e

and in the caller, handle that separately from success is False: don't append it to the conversation, don't count it against guardrail_max_retries, and surface it with its own message. If you'd rather not raise, a third state on GuardrailResult (something like errored: bool) would carry the same information without changing the return type.

Either way the point is that "I couldn't check" stops being reported as "the output is bad".

Additional context

I found this while looking at how guardrail components get their judgement, across ten agent frameworks pinned to specific commits. One related thing worth mentioning, though I'd rather keep it out of this issue as a separate question: the guardrail is constructed with the same BaseLLM instance as the agent it judges (task.py:393, agent/core.py:1849, lite_agent.py:382). That matters here because it makes the scenario above more likely, not less. The same outage that breaks the guardrail is hitting the agent's own generation, so the three retries are being spent while both halves are down.

For what it's worth, of the ten frameworks I looked at, six ship a guardrail of their own and four of those keep it off the subject's client. LlamaFirewall's scanners build their own client, Guardrails AI takes the LLM as an injected
callable, LangChain's safety middlewares don't call a model at all. Happy to open that as its own feature request if you'd find it useful, but this issue is just about the error handling.

I'm writing this work up and will report whatever you say accurately, including if I've misread something. Happy to share the reproduction script.

Environment

• crewAI at commit 6491f5a (lib/crewai/src/crewai)
• Reproduced by extracting LLMGuardrail.call from that commit and running it against a _validate_output that raises, so no API key or network is needed
• Python 3.12

Steps to Reproduce

No API key or network needed. This isolates LLMGuardrail.call and makes the LLM call fail.

1. pip install crewai==1.15.1
2. Save the snippet from the Code snippets box below as repro.py
3. python repro.py
4. Both the outage case and a genuine guardrail violation print (False, ...). The caller has no way to tell them apart.

Expected behavior

An exception raised while the guardrail is running should be distinguishable from a validation result.

If the guardrail could not run, the caller should be able to handle that on its own terms: not treat it as evidence about the output, not feed the error text back into the agent's conversation, not spend a retry on it, and surface an
error that names the real cause.

Right now "couldn't check" and "check says it's bad" are the same value, so the caller does the wrong thing with the first one.

Screenshots/Code snippets

from unittest.mock import patch

from crewai.tasks.llm_guardrail import LLMGuardrail, LLMGuardrailResult
from crewai.tasks.task_output import TaskOutput

out = TaskOutput(description="d", agent="a", raw="the agent's answer")
g = LLMGuardrail(description="must be under 100 words", llm=None)

1. the LLM call fails

with patch.object(LLMGuardrail, "_validate_output",
side_effect=RuntimeError("litellm.APIConnectionError: provider unavailable")):
print("outage ->", g(out))

2. a genuine guardrail violation

class FakeOut:
pydantic = LLMGuardrailResult(valid=False, feedback="too long by 40 words")

with patch.object(LLMGuardrail, "_validate_output", return_value=FakeOut()):
print("real violation ->", g(out))

3. control: a passing output is still distinguishable

class PassOut:
pydantic = LLMGuardrailResult(valid=True, feedback=None)

with patch.object(LLMGuardrail, "_validate_output", return_value=PassOut()):
print("passes ->", g(out))

The source it exercises, lib/crewai/src/crewai/tasks/llm_guardrail.py:110-119:

try:
result = self._validate_output(task_output)
if not isinstance(result.pydantic, LLMGuardrailResult):
raise ValueError("The guardrail result is not a valid pydantic model")
if result.pydantic.valid:
return True, task_output.raw
return False, result.pydantic.feedback
except Exception as e:
return False, f"Error while validating the task output: {e!s}"

Operating System

Windows 11

Python Version

3.12

crewAI Version

1.15.1 (commit 6491f5a)

crewAI Tools Version

Not used. The reproduction only imports crewai.tasks.llm_guardrail and crewai.tasks.task_output.

Virtual Environment

Venv

Evidence

Output of the snippet:

outage -> (False, 'Error while validating the task output: litellm.APIConnectionError: provider unavailable')
real violation -> (False, 'too long by 40 words')
passes -> (True, "the agent's answer")

The first two are the same shape. The check itself works, since a passing output is still distinguishable. The problem is only that a failure to run is reported as a failure of the output.

What the caller then does with that False, in lib/crewai/src/crewai/agent/core.py:1865-1878:

if not guardrail_result.success:
if retry_count >= self.guardrail_max_retries:
raise ValueError(
f"Agent's guardrail failed validation after {self.guardrail_max_retries} retries. "
f"Last error: {guardrail_result.error}"
)
executor._append_message_to_state(
guardrail_result.error or "Guardrail validation failed", role="user"
)
output = self._execute_and_build_output(executor, inputs, response_format)
return self._process_kickoff_guardrail(output=output, ...)

lib/crewai/src/crewai/task.py:1258 onward has the same shape.

So during a provider incident:

1. The raw exception text is appended into the agent's conversation as a user turn. Provider error bodies sometimes echo parts of the request, so whatever the provider said ends up back in the model's context.
2. The agent is re-executed in full, up to guardrail_max_retries (default 3, agent/core.py:318). Three extra complete agent runs during an outage, each likely to hit the same error.
3. It finally raises ValueError: Agent's guardrail failed validation after 3 retries, which tells whoever is on call that validation failed when the provider was actually down.

To be clear about what this is not: nothing bad gets through. It fails closed, and I checked that specifically before filing. This is a correctness and diagnosability problem, not a bypass.

Possible Solution

Give the exception path its own case instead of borrowing the "invalid" one.

class GuardrailExecutionError(Exception):
"""The guardrail could not run. Not a statement about the output."""

in LLMGuardrail.call

except Exception as e:
raise GuardrailExecutionError(str(e)) from e

Then in _process_kickoff_guardrail and the equivalent in task.py, handle that separately from success is False: don't append it to the conversation, don't count it against guardrail_max_retries, and raise something that names the real
cause.

If raising is too disruptive for existing users, a third state on GuardrailResult carries the same information without changing the return type:

class GuardrailResult(BaseModel):
success: bool
errored: bool = False # the guardrail could not run
result: Any | None = None
error: str | None = None

Either shape works. The point is that "I couldn't check" stops being reported as "the output is bad".

Additional context

I found this while looking at how guardrail components obtain their judgement, across ten agent frameworks pinned to specific commits.

One related thing, which I'd rather keep out of this issue as a separate question but which matters here: the guardrail is constructed with the same BaseLLM instance as the agent it judges (task.py:393, agent/core.py:1849,
lite_agent.py:382). That makes the situation above more likely rather than less, because the same outage that breaks the guardrail is also hitting the agent's own generation. The three retries get spent while both halves are down.

Of the ten frameworks I looked at, six ship a guardrail of their own and four of those keep it off the subject's client. LlamaFirewall's scanners build their own client, Guardrails AI takes the LLM as an injected callable that no
validator touches, and LangChain's safety middlewares don't call a model at all. Happy to open that as its own feature request if it would be useful, but this issue is only about the error handling.

I'm writing this work up and will report whatever you say accurately, including if I've misread something.