aws / aws/bedrock-agentcore-sdk-python
OTEL context token detached across asyncio/thread boundary in memory client + Strands session_manager
- Dominant language
- Python
- Stars
- 761
- Forks
- 147
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 7
Description
## Summary
When using `bedrock-agentcore` (1.6.2) with the Strands integration inside an AgentCore runtime, the OpenTelemetry runtime logs a steady stream of `Failed to detach context` errors:
```
ERROR [opentelemetry.context] [__init__.py:157] - Failed to detach context
Traceback (most recent call last):
File "/var/task/opentelemetry/context/__init__.py", line 155, in detach
_RUNTIME_CONTEXT.detach(token)
File "/var/task/opentelemetry/context/contextvars_context.py", line 53, in detach
self._current_context.reset(token)
ValueError: at 0x...> was created in a different Context
```
The errors are logged from spans owned by:
- `bedrock_agentcore.memory.client` (around `create_event` / `client.py:484`)
- `bedrock_agentcore.memory.integrations.strands.session_manager` (`session_manager.py:406`, around `Created agent: default in session: …`)
Functionally tracing still works (events are created, agents start), but ERROR-level log spam pollutes CloudWatch and obscures real failures.
## Root cause (best read)
The SDK calls `opentelemetry.context.attach(ctx)` to set span context, then `opentelemetry.context.detach(token)` to restore. Per PEP 567, every `asyncio.Task` (and every threadpool worker) gets its own copy of the contextvars state, so a token created in one execution context is not valid to `reset()` in another.
This shows up specifically when:
1. A boto3 sync call (e.g., `CreateEvent`) is invoked from inside an `async` method and ends up resuming on a different thread / task than the one that called `attach`.
2. A Strands session manager hook (e.g. `Created agent` event) attaches in one task and detaches after an `await` resumes on another task.
## Repro environment
- `bedrock-agentcore==1.6.2`
- `bedrock-agentcore-starter-toolkit==0.3.5`
- AgentCore-managed runtime (Lambda-style), `aws-opentelemetry-distro` enabled
- Strands `Agent` constructed with `AgentCoreMemorySessionManager`
- Concurrent module-builder agents launched from a `ThreadPoolExecutor`
## Expected
`detach()` should not raise (or the SDK should not call it across context boundaries). Either:
- Use `with use_span(...)` / `set_value` patterns that don't require explicit `detach`, or
- Capture the originating context and only detach if the current context still owns the token, or
- Wrap the detach in a try/except `ValueError` since a stale token can be safely ignored.
## Workaround consumers can apply
```python
from opentelemetry import context as _ctx
_orig_detach = _ctx.detach
def _safe_detach(token):
try:
_orig_detach(token)
except ValueError:
pass
_ctx.detach = _safe_detach
```
…installed before importing `bedrock_agentcore`. This is purely cosmetic and should live in the SDK instead.
## Suggested fix
In `bedrock_agentcore.memory.client` and `bedrock_agentcore.memory.integrations.strands.session_manager`, wrap each `context.detach(token)` call with a `try/except ValueError` (or refactor to use `opentelemetry.trace.use_span` so no manual token management is needed across `await` boundaries).
Contributor guide
Assessment
This issue has not been assessed yet.