New Relic Python agent breaks SigV4 signing for `bedrock-agentcore:InvokeAgentRuntime` (13.x)

New Relic Python agent breaks SigV4 signing for bedrock-agentcore:InvokeAgentRuntime (13.x)

Upgraded NR Python lib from 11.5 to 13.5 and it completely broke agentcore! Here is the indepth Claude (Opus 4.8) debugging/analysis - thanks!

Summary

With the Python agent active and connected (distributed tracing enabled), calls to
Amazon Bedrock AgentCore invoke_agent_runtime fail with:

An error occurred (AccessDeniedException) when calling the InvokeAgentRuntime operation:
The request signature we calculated does not match the signature you provided.
Check your AWS Secret Access Key and signing method.

Root cause: the agent injects its distributed-tracing headers (traceparent, tracestate,
newrelic) into the outbound HTTP request after botocore has already SigV4-signed it.
bedrock-agentcore:InvokeAgentRuntime is (as far as we can tell) the only AWS API whose
SigV4 SignedHeaders include traceparent — it is a modelled request header — so mutating
/ duplicating traceparent post-signing changes the canonical request the service reconstructs,
and signature verification fails.

Every other AWS API we use (bedrock-runtime, DynamoDB, OpenSearch, S3, …) does not sign
traceparent, so the injected header is harmless there. This is why only AgentCore calls fail
while all other AWS calls on the same process/credentials succeed.

Impact

• Regression appears when upgrading the agent 11.5.0 → 13.5.0 (all other deps, including
  boto3 1.43.87, held constant — see below).
• Fails deterministically only with a connected agent (monitor_mode = true), so it does not
  reproduce with the agent disabled/local dev.
• Not credential- or region-related: the same task/credentials sign every other AWS request fine.

Environment / libraries used to reproduce

Component	"Works"	"Fails"
newrelic	11.5.0	13.5.0
boto3	1.42.54	1.43.87
botocore	1.42.97	1.43.87
Python	3.14.3	3.14.3
Service	bedrock-agentcore (invoke_agent_runtime)	same
Distributed tracing	enabled, agent connected	enabled, agent connected

Isolation: we upgraded every dependency in our image except newrelic (kept boto3 at
1.43.87) and the failure disappeared. Reverting only newrelic 13.5.0 → 11.5.0 also fixes it.
botocore 1.42.97 vs 1.43.87 produce byte-identical SigV4 canonical requests for this call, so
the SDK is not involved.

The signed request

botocore signs the modelled traceparent header for this operation. Captured SignedHeaders
for invoke_agent_runtime:

SignedHeaders=host;traceparent;x-amz-date;x-amzn-bedrock-agentcore-runtime-session-id

The caller supplies traceParent (W3C format, e.g. 00-<32 hex>-<16 hex>-01) as an operation
parameter, which botocore serialises to the traceparent header and includes in the signature.

Where the agent injects (source references, 13.5.0)

newrelic/hooks/external_httplib.py — httplib_endheaders_wrapper (wraps
HTTPConnection.endheaders) adds NR headers to the already-built request just before send:

if not skip_headers and hasattr(tracer, "generate_request_headers"):
    outgoing_headers = tracer.generate_request_headers(transaction)
    for header_name, header_value in outgoing_headers:
        connection.putheader(header_name, header_value)

• newrelic/hooks/external_httplib2.py / external_urllib3.py route botocore's urllib3
  connections through the same endheaders instrumentation (sets _nr_library_info).
• newrelic/api/header_mixin.py → generate_request_headers → transaction.insert_distributed_trace_headers(...)
  emits traceparent / tracestate / newrelic when distributed tracing is enabled.

Because endheaders runs after botocore signs the request, the injected traceparent
is added on top of the one that was already signed → signature mismatch.

Relevant changes 11.5.0 → 13.5.0 (outbound-header path)

• newrelic/api/external_trace.py: ExternalTrace(CatHeaderMixin) → ExternalTrace(HeaderMixin).
• New newrelic/api/header_mixin.py replaces newrelic/api/cat_header_mixin.py;
  generate_request_headers dropped the legacy CAT branch and now emits W3C DT headers only.
• newrelic/hooks/external_httplib.py: the _nr_skip_headers detection narrowed from
  {"NEWRELIC", "X-NEWRELIC-ID", "X-NEWRELIC-TRANSACTION"} to {"NEWRELIC"}.

Steps to reproduce

1. Install newrelic==13.5.0, boto3==1.43.87 on Python 3.14.
2. Run under a connected agent with distributed tracing enabled
   (NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true, real license, monitor_mode=true), inside a
   web/background transaction.
3. Call:

   client = boto3.client("bedrock-agentcore", region_name="us-east-1")
   client.invoke_agent_runtime(
       agentRuntimeArn="arn:aws:bedrock-agentcore:us-east-1:<acct>:runtime/<id>/runtime-endpoint/PROD",
       payload=b'{"prompt": "hello"}',
       traceParent="00-" + "a"*32 + "-" + "b"*16 + "-01",
   )
   

4. Observe AccessDeniedException … signature does not match. With newrelic==11.5.0 (same
   boto3), the same call succeeds.

Note: the injection only manifests with a connected agent; a disabled/developer-mode agent
does not generate DT headers, so it will not reproduce.

Expected behaviour

The agent should not inject distributed-tracing headers into AWS SigV4-signed requests (it
already avoids breaking signing for other AWS services). For bedrock-agentcore, whose signed
headers include traceparent, injecting/overwriting traceparent after signing must be avoided.