after_llm_call hooks never run on acall(): eight native provider handlers skip _invoke_after_llm_call_hooks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Description
after_llm_call hooks never run on acall(). The native providers invoke
_invoke_after_llm_call_hooks only from their sync handlers, so a hook registered to
redact, rewrite or audit a response is silently skipped for every async direct LLM call — the
unmodified response reaches the caller and nothing is logged.
The gap is per-handler, not per-provider. Coverage on main (3266932):
| provider | handler | sync | async |
|---|---|---|---|
| openai | _handle_responses / _ahandle_responses |
yes (:1068) |
no |
| openai | _handle_streaming_responses / _ahandle_streaming_responses |
yes (:1376) |
no |
| openai | _handle_completion / _ahandle_completion |
yes (:2008) |
no |
| openai | _handle_streaming_completion / _ahandle_streaming_completion |
yes (:2298) |
no |
| anthropic | _handle_completion / _ahandle_completion |
yes (:1104) |
no |
| anthropic | _handle_streaming_completion / _ahandle_streaming_completion |
yes (:1307) |
no |
| bedrock | _handle_converse / _ahandle_converse |
yes (:823) |
no |
| bedrock | _handle_streaming_converse / _ahandle_streaming_converse |
no | yes (:1772) |
The bedrock row is the giveaway: within one file, _ahandle_streaming_converse calls the hook
and _ahandle_converse does not, and the sync/async polarity is inverted relative to every
other row. That is not a design decision about async — it is eight handlers each missing the
call their twin makes.
Reproduction
No network. Stub the SDK client and compare call() with acall():
from crewai.hooks.llm_hooks import register_after_llm_call_hook
calls = []
def hook(ctx):
calls.append(ctx.response)
return "REDACTED BY HOOK"
register_after_llm_call_hook(hook)
llm = AnthropicCompletion(model="claude-sonnet-4-5", thinking={"type": "enabled", "budget_tokens": 1024},
max_tokens=2048, stream=False)
llm._client = SimpleNamespace(messages=SyncStub()) # returns one thinking + one text block
llm._async_client = SimpleNamespace(messages=AsyncStub()) # same payload
print(llm.call("hi"))
print(asyncio.run(llm.acall("hi")))
On main:
anthropic sync : 'REDACTED BY HOOK' hook fired: 1
anthropic async: 'the model said something sensitive' hook fired: 0
openai sync : 'REDACTED BY HOOK' hook fired: 1
openai async: 'the model said something sensitive' hook fired: 0
The hook is never even invoked on the async path, so a hook that only observes (logging,
auditing, metrics) also silently records nothing.
Second defect in the same code: anthropic loses thinking blocks on the async path
While mapping the above I found _ahandle_completion and _ahandle_streaming_completion never
call _extract_thinking_block, so _previous_thinking_blocks stays empty on the async path.
The same probe:
anthropic sync thinking kept: [{'type': 'thinking', 'thinking': 'internal reasoning', 'signature': 'sig-abc'}]
anthropic async thinking kept: []
_format_messages_for_anthropic:851 replays self._previous_thinking_blocks into the
assistant message on the next turn when thinking is enabled. With the async path leaving that
list empty, the signed thinking block is dropped and the follow-up request loses the reasoning
context the sync path preserves. test_anthropic_thinking_blocks_preserved_across_turns
(tests/llms/anthropic/test_anthropic.py:633) pins this behaviour precisely — for call() only.
Why this matters
after_llm_call is the documented interception point for exactly the things that must not be
skippable: redacting PII before a response is stored or displayed, enforcing an output policy,
rewriting a response, and audit logging. A guardrail that works under call() and silently
stops working under acall() is worse than one that never worked, because the failure is
invisible: no exception, no warning, and the response looks plausible.
Async is not an edge path — acall() is what kickoff_async, async flows and any concurrent
usage reach. Moving a working crew from sync to async silently disables every registered
after_llm_call hook.
Expected behaviour
acall() runs after_llm_call hooks exactly as call() does, and anthropic retains thinking
blocks on both paths.
Fix
Add the missing _invoke_after_llm_call_hooks call to each of the eight handlers, mirroring the
placement its twin already uses (after _emit_call_completed_event, and behind the same
isinstance(result, str) guard the sync streaming handler uses so structured tool-call payloads
are not stringified), plus the two _extract_thinking_block loops in anthropic's async handlers.
_invoke_after_llm_call_hooks already returns the response unchanged when from_agent is not
None, so the agent-driven path (which dispatches POST_MODEL_CALL separately via
agent_utils._setup_after_llm_call_hooks) is unaffected — no double dispatch.
Note on scope: azure and gemini call the hook on neither path. That is a symmetric
gap — the provider does not support the hook at all — rather than sync/async drift, so I left it
out of this fix; it needs a separate decision about whether those providers should support
direct-call hooks. Happy to follow up if wanted.
I have a PR ready with the fix plus 11 tests (sync baselines, async regressions, and negative
controls asserting an unhooked response passes through untouched).
Environment
- crewAI
main@3266932 - Python 3.12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the eight async handlers named in the issue and compare each with its synchronous twin, then inspect anthropic’s _ahandle_completion and _ahandle_streaming_completion paths. Run tests/llms/anthropic/test_anthropic.py, especially test_anthropic_thinking_blocks_preserved_across_turns; done means async hooks fire like sync hooks and thinking blocks remain preserved without altering unhooked responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100