Anthropic instrumentation drops token/content capture for messages.with_raw_response.create (breaks langchain-anthropic / ChatAnthropic)

Summary

When the Anthropic SDK is called through client.messages.with_raw_response.create(...), the Python agent records an LlmChatCompletionSummary (vendor = "anthropic") but with no token/usage data — response.usage.* is never populated, so AI Monitoring reports 0 tokens (and the completion content is not captured). A direct client.messages.create(...) call captures tokens and content correctly under identical conditions.

This matters in practice because langchain-anthropic's ChatAnthropic calls the SDK exclusively via with_raw_response, so LangChain / LangGraph apps on Anthropic silently lose token + completion-content capture even though the summary events are recorded.

Environment

• newrelic==13.5.0
• anthropic==1.6.0
• langchain-anthropic==1.7.2, langchain-core==1.6.3 (the repro needs only langchain-anthropic)
• Python 3.14
• ai_monitoring.enabled = true, ai_monitoring.record_content.enabled = true, ai_monitoring.streaming.enabled = true (default)

Reproduction

Run under newrelic-admin run-program (so import order is not a factor). Same process, same model; the two paths are tagged so they're separable in NRDB:

import os
os.environ["ANTHROPIC_BASE_URL"] = "https://api.anthropic.com"

import newrelic.agent          # already initialized by newrelic-admin
from anthropic import Anthropic
from langchain_anthropic import ChatAnthropic

MODEL = "claude-haiku-4-5-20251001"
_sdk = Anthropic()
_lc = ChatAnthropic(model=MODEL, max_tokens=64)

@newrelic.agent.background_task(name="repro_sdk")
def via_sdk():
    newrelic.agent.add_custom_attribute("llm.path", "sdk")
    _sdk.messages.create(model=MODEL, max_tokens=64,
                         messages=[{"role": "user", "content": "say hi in 3 words"}])

@newrelic.agent.background_task(name="repro_chatanthropic")
def via_langchain():
    newrelic.agent.add_custom_attribute("llm.path", "chatanthropic")
    _lc.invoke("say hi in 3 words")

for _ in range(8):
    via_sdk(); via_langchain()
newrelic.agent.shutdown_agent(timeout=30)

newrelic-admin run-program python repro.py

Then query the events:

SELECT count(*) AS summaries,
       filter(count(*), WHERE `response.usage.total_tokens` IS NOT NULL) AS with_usage,
       sum(`response.usage.total_tokens`) AS total_tokens
FROM LlmChatCompletionSummary FACET `llm.path`, vendor SINCE 30 minutes ago

Both paths record vendor = anthropic summaries; only the direct-SDK path carries usage. (8 calls were issued per path; the ratio is the point — 100% of SDK-path summaries carry response.usage, 0% of ChatAnthropic-path summaries do.)

Root cause

newrelic/hooks/mlmodel_anthropic.py (instrument_anthropic_messages) wraps Messages.create / Messages.stream (and the async variants). langchain-anthropic calls self._client.messages.with_raw_response.create(**payload) (langchain_anthropic/chat_models.py; there is also a beta.messages.with_raw_response.create variant).

In the SDK, messages.with_raw_response.create is to_raw_response_wrapper(messages.create), so the New Relic wrapper on Messages.create does fire (that is why a vendor=anthropic summary is recorded at all) — but the object returned to the wrapper is the raw API response (LegacyAPIResponse / APIResponse), not a parsed Message. The hook then does:

usage = getattr(response, "usage", None)          # mlmodel_anthropic.py (~L168 sync / ~L256 async)
...
input_tokens=getattr(usage, "input_tokens", None) if usage else None,
output_tokens=getattr(usage, "output_tokens", None) if usage else None,

On a raw-response wrapper usage is None, so no response.usage.* is recorded, and the assistant content isn't parsed from the raw response either.

What this is not

• Reproduced under newrelic-admin run-program, so it is not the known LangChain import-priority issue.
• Reproduces with a bare ChatAnthropic().invoke() — no create_agent, no LangGraph — so it is below the agent layer; LangChain tool/agent instrumentation itself works (LlmTool events are recorded fine in a fuller app).
• The direct SDK path works identically in the same process, isolating the with_raw_response entrypoint as the only difference.

Suggested fix

Handle the raw-response case in the Anthropic hook — e.g. instrument MessagesWithRawResponse.create (and the beta variants), or, when the wrapped call returns a raw API response, .parse() it (or read the underlying payload) before extracting usage/content.

Start in newrelic/hooks/mlmodel_anthropic.py at instrument_anthropic_messages and compare the sync and async Messages.create paths with the raw-response entry points described in the issue, including beta variants. Re-run the supplied SDK and ChatAnthropic reproduction and query LlmChatCompletionSummary events; done means raw-response calls capture usage tokens and completion content without regressing direct SDK calls.