aws / aws/bedrock-agentcore-sdk-python

gen_ai spans (invoke_agent / execute_tool) omit calling user identity and client IP — add enduser.id / client.address

Open
#577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
761
Forks
147
Avg merge
1d 23h
Merged PRs (30d)
7

Description

## Summary

The GenAI observability spans emitted for an AgentCore-hosted agent
(`invoke_agent`, `execute_tool`, `chat`, `POST /invocations`) contain **no identity
of the caller who invoked the agent and no client IP address**. The invoking IAM
principal / end user and the real client address are absent from every span and from
the correlated GenAI content log records. This makes it impossible to attribute an
agent session (and its tool calls) to *who* triggered it using CloudWatch
observability alone.

## Observed spans — no identity/network attributes

For a full invocation I inspected every span (verified live). None carry any
caller-identity or client-IP attribute:

- `invoke_agent Strands Agents` — has `gen_ai.*`, `gen_ai.usage.*`, `harness.id`,
`session.id`; **no** user or client address.
- `execute_tool ` — has `gen_ai.tool.*`, `session.id`; **no** user or client address.
- `chat` / `chat ` — model attrs only.
- `POST /invocations` (LOCAL_ROOT) — has only **internal** transport values
(`net.peer.ip = 127.0.0.1`, `http.host = 127.0.0.1:8080`,
`http.url = http://cell01.us-east-2.prod.arp.kepler-analytics.aws.dev/invocations`);
the external caller's IP/identity is not present.

So neither the identity nor the true client address of the `InvokeAgentRuntime`
caller is propagated onto the spans.

## Relevant OpenTelemetry semantic conventions (the schema gap)

These spans should be able to carry the caller identity and client address using
existing OTel conventions:

**End user identity** — https://opentelemetry.io/docs/specs/semconv/registry/attributes/enduser/
| Attribute | Type | Description |
|---|---|---|
| `enduser.id` | string | Unique identifier of an end user (username/email/…) — PII |
| `enduser.pseudo.id` | string | Pseudonymous (non-PII) end-user identifier |
| `user.roles` | string[] | (replaces deprecated `enduser.role`) |

**Client / network** — https://opentelemetry.io/docs/specs/semconv/registry/attributes/client/
| Attribute | Stability | Description |
|---|---|---|
| `client.address` | Stable | Client address behind any intermediaries (the real caller IP) |
| `client.port` | Stable | Client port |
| `user_agent.original` | Stable | Client user agent |

None of these are populated on the AgentCore/Strands GenAI spans.

## Request

Stamp the invoking caller's identity and client address onto the agent spans
(at minimum the root `invoke_agent` / `POST /invocations` span, ideally inherited by
child `execute_tool` spans), using the OTel conventions above:

- `enduser.id` (or `enduser.pseudo.id` for privacy) — the invoking principal / end
user (e.g. from the `InvokeAgentRuntime` caller identity, or from the
`X-Amzn-Bedrock-AgentCore-Runtime-User-Id` header when
`InvokeAgentRuntimeForUser` is used).
- `client.address` (+ `client.port`, `user_agent.original`) — the real client that
called `InvokeAgentRuntime`, not the internal `127.0.0.1` loopback currently on
`POST /invocations`.

Gating these behind an opt-in (they are sensitive/PII) is fine.

## Why this can't be worked around today

The caller identity and real client IP **do** exist in the `InvokeAgentRuntime`
**CloudTrail data event** (`userIdentity`, `sourceIPAddress`, `userAgent`) — but that
event has `requestParameters: null` and shares **no** join key with the spans
(`session.id` is not in CloudTrail; `requestID`/`eventID` are not in any span). A full
field-value intersection of the two returns only the region. So identity cannot be
attached to a span/session from telemetry alone — it must be captured client-side. Put
another way: the data exists in CloudTrail but is **unjoinable** to the spans, which is
exactly why stamping `enduser.id` / `client.address` onto the spans is needed.

## Environment

- Amazon Bedrock AgentCore managed harness (Strands agent, `amazon.nova-lite-v1:0`),
region us-east-2.
- Spans from `strands.telemetry.tracer`, `telemetry.auto.version = 0.18.0-aws`,
delivered to CloudWatch `aws/spans` (Transaction Search enabled).
- Verified: no `enduser.*`, `client.address`, or external `sourceIPAddress` on any
span; `POST /invocations` shows only internal loopback transport values.

## Steps to reproduce the behavior

**Prereqs:** an AgentCore-hosted agent (e.g. managed harness running Strands) with at
least one tool, and CloudWatch Transaction Search enabled (spans in `aws/spans`).

1. **Invoke the agent** with a fresh session id, from a client with an identifiable AWS
principal (assumed-role/SSO) and a routable public IP:

```python import boto3, uuid c = boto3.client("bedrock-agentcore", region_name="") sid = "repro-" + uuid.uuid4().hex resp = c.invoke_harness( # or invoke_agent_runtime
harnessArn="",
runtimeSessionId=sid,
messages=[{"role": "user", "content": [{"text":
"Use the tool to get info, then summarize it in one sentence."}]}], ) for _ in resp.get("stream", []):
pass print("session:", sid)

2. Pull the spans for that session from CloudWatch Logs group  aws/spans  and dump
every span's attributes:
limit=5000, filterPattern=f'"{sid}"') for e in r["events"]:
m = json.loads(e["message"])
if m.get("name"):
print(m["name"], "→", sorted(m.get("attributes", {})))
3. Inspect the root span  POST /invocations  and the  invoke_agent  /  execute_tool 
spans for any caller-identity or client-address attribute.

Result: no span contains  enduser.id ,  enduser.pseudo.id ,  client.address ,
 client.port , or  user_agent.original . The only network values are internal
( POST /invocations :  net.peer.ip = 127.0.0.1 ,  http.host = 127.0.0.1:8080 ); the
external caller's IP and identity never appear. (The real caller IP + IAM principal are
present only in the  InvokeAgentRuntime  CloudTrail data event, which shares no join
key with these spans.)

Expected behavior

The invoking caller's identity and real client address should be recorded on the agent
spans using the standard OpenTelemetry conventions, so a session/trace can be attributed
to who triggered it from CloudWatch alone:

•  enduser.id  (or  enduser.pseudo.id  for a non-PII identifier) = the invoking
principal / end user (from the  InvokeAgentRuntime  caller identity, or from
 X-Amzn-Bedrock-AgentCore-Runtime-User-Id  when  InvokeAgentRuntimeForUser  is used).
•  client.address  (+  client.port ,  user_agent.original ) = the real client that
called  InvokeAgentRuntime  — not the internal  127.0.0.1  loopback.

At minimum these should appear on the root  invoke_agent  /  POST /invocations  span and
be inheritable by child  execute_tool  spans. Gating them behind an opt-in (PII) is fine.

Additional context

• This is a service-level gap: the agent runs behind the AgentCore runtime's internal
proxy, so the agent/tracer only sees  127.0.0.1 . Only the managed runtime/service can
inject the true external caller identity/IP — hence filed here rather than against the
Strands SDK.
• Cannot be worked around from telemetry: the caller identity + real IP exist in the
 InvokeAgentRuntime  CloudTrail data event ( userIdentity ,  sourceIPAddress ,
 userAgent ), but that event has  requestParameters: null  and shares no per-invocation
key with the spans — a full field-value intersection of the two returns only the region.
(Companion request: "Add  runtimeSessionId  to the  InvokeAgentRuntime  CloudTrail data
event" — the two together close the attribution gap from either side.)
• OTel references: end-user attrs
https://opentelemetry.io/docs/specs/semconv/registry/attributes/enduser/ ·
client attrs https://opentelemetry.io/docs/specs/semconv/registry/attributes/client/
• Env: managed harness, Strands agent ( amazon.nova-lite-v1:0 ), region us-east-2;
spans from  strands.telemetry.tracer ,  telemetry.auto.version = 0.18.0-aws .

Contributor guide

Open the contributing guide

Research direction

Start by tracing the SDK's telemetry entry points around strands.telemetry.tracer and the managed harness spans, then verify whether caller identity and client address are available to this repository or only to the AgentCore service. Compare the POST /invocations, invoke_agent, and execute_tool spans in CloudWatch, and consider the OpenTelemetry enduser and client conventions; done means a supported design or implementation can attribute spans without relying on unjoinable CloudTrail data.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.