crewAIInc / crewAIInc/crewAI

after_llm_call hooks never run on acall(): eight native provider handlers skip _invoke_after_llm_call_hooks

Open
#6,736 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

no-issue-activity
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

after_llm_call hooks never run on acall(). The native providers invoke
_invoke_after_llm_call_hooks only from their sync handlers, so a hook registered to
redact, rewrite or audit a response is silently skipped for every async direct LLM call — the
unmodified response reaches the caller and nothing is logged.

The gap is per-handler, not per-provider. Coverage on main (3266932):

provider handler sync async
openai _handle_responses / _ahandle_responses yes (:1068) no
openai _handle_streaming_responses / _ahandle_streaming_responses yes (:1376) no
openai _handle_completion / _ahandle_completion yes (:2008) no
openai _handle_streaming_completion / _ahandle_streaming_completion yes (:2298) no
anthropic _handle_completion / _ahandle_completion yes (:1104) no
anthropic _handle_streaming_completion / _ahandle_streaming_completion yes (:1307) no
bedrock _handle_converse / _ahandle_converse yes (:823) no
bedrock _handle_streaming_converse / _ahandle_streaming_converse no yes (:1772)

The bedrock row is the giveaway: within one file, _ahandle_streaming_converse calls the hook
and _ahandle_converse does not, and the sync/async polarity is inverted relative to every
other row. That is not a design decision about async — it is eight handlers each missing the
call their twin makes.

Reproduction

No network. Stub the SDK client and compare call() with acall():

from crewai.hooks.llm_hooks import register_after_llm_call_hook

calls = []
def hook(ctx):
    calls.append(ctx.response)
    return "REDACTED BY HOOK"
register_after_llm_call_hook(hook)

llm = AnthropicCompletion(model="claude-sonnet-4-5", thinking={"type": "enabled", "budget_tokens": 1024},
                          max_tokens=2048, stream=False)
llm._client       = SimpleNamespace(messages=SyncStub())    # returns one thinking + one text block
llm._async_client = SimpleNamespace(messages=AsyncStub())   # same payload

print(llm.call("hi"))
print(asyncio.run(llm.acall("hi")))

On main:

anthropic sync : 'REDACTED BY HOOK'                     hook fired: 1
anthropic async: 'the model said something sensitive'   hook fired: 0
openai    sync : 'REDACTED BY HOOK'                     hook fired: 1
openai    async: 'the model said something sensitive'   hook fired: 0

The hook is never even invoked on the async path, so a hook that only observes (logging,
auditing, metrics) also silently records nothing.

Second defect in the same code: anthropic loses thinking blocks on the async path

While mapping the above I found _ahandle_completion and _ahandle_streaming_completion never
call _extract_thinking_block, so _previous_thinking_blocks stays empty on the async path.
The same probe:

anthropic sync  thinking kept: [{'type': 'thinking', 'thinking': 'internal reasoning', 'signature': 'sig-abc'}]
anthropic async thinking kept: []

_format_messages_for_anthropic:851 replays self._previous_thinking_blocks into the
assistant message on the next turn when thinking is enabled. With the async path leaving that
list empty, the signed thinking block is dropped and the follow-up request loses the reasoning
context the sync path preserves. test_anthropic_thinking_blocks_preserved_across_turns
(tests/llms/anthropic/test_anthropic.py:633) pins this behaviour precisely — for call() only.

Why this matters

after_llm_call is the documented interception point for exactly the things that must not be
skippable: redacting PII before a response is stored or displayed, enforcing an output policy,
rewriting a response, and audit logging. A guardrail that works under call() and silently
stops working under acall() is worse than one that never worked, because the failure is
invisible: no exception, no warning, and the response looks plausible.

Async is not an edge path — acall() is what kickoff_async, async flows and any concurrent
usage reach. Moving a working crew from sync to async silently disables every registered
after_llm_call hook.

Expected behaviour

acall() runs after_llm_call hooks exactly as call() does, and anthropic retains thinking
blocks on both paths.

Fix

Add the missing _invoke_after_llm_call_hooks call to each of the eight handlers, mirroring the
placement its twin already uses (after _emit_call_completed_event, and behind the same
isinstance(result, str) guard the sync streaming handler uses so structured tool-call payloads
are not stringified), plus the two _extract_thinking_block loops in anthropic's async handlers.

_invoke_after_llm_call_hooks already returns the response unchanged when from_agent is not
None, so the agent-driven path (which dispatches POST_MODEL_CALL separately via
agent_utils._setup_after_llm_call_hooks) is unaffected — no double dispatch.

Note on scope: azure and gemini call the hook on neither path. That is a symmetric
gap — the provider does not support the hook at all — rather than sync/async drift, so I left it
out of this fix; it needs a separate decision about whether those providers should support
direct-call hooks. Happy to follow up if wanted.

I have a PR ready with the fix plus 11 tests (sync baselines, async regressions, and negative
controls asserting an unhooked response passes through untouched).

Environment
  • crewAI main @ 3266932
  • Python 3.12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the eight async handlers named in the issue and compare each with its synchronous twin, then inspect anthropic’s _ahandle_completion and _ahandle_streaming_completion paths. Run tests/llms/anthropic/test_anthropic.py, especially test_anthropic_thinking_blocks_preserved_across_turns; done means async hooks fire like sync hooks and thinking blocks remain preserved without altering unhooked responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.