aws / aws/bedrock-agentcore-sdk-python
OTEL context token detached across asyncio/thread boundary in memory client + Strands session_manager
- Ngôn ngữ chính
- Python
- Star
- 764
- Fork
- 148
- Merge trung bình
- 1 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 7
Mô tả
## Summary
When using `bedrock-agentcore` (1.6.2) with the Strands integration inside an AgentCore runtime, the OpenTelemetry runtime logs a steady stream of `Failed to detach context` errors:
```
ERROR [opentelemetry.context] [__init__.py:157] - Failed to detach context
Traceback (most recent call last):
File "/var/task/opentelemetry/context/__init__.py", line 155, in detach
_RUNTIME_CONTEXT.detach(token)
File "/var/task/opentelemetry/context/contextvars_context.py", line 53, in detach
self._current_context.reset(token)
ValueError: at 0x...> was created in a different Context
```
The errors are logged from spans owned by:
- `bedrock_agentcore.memory.client` (around `create_event` / `client.py:484`)
- `bedrock_agentcore.memory.integrations.strands.session_manager` (`session_manager.py:406`, around `Created agent: default in session: …`)
Functionally tracing still works (events are created, agents start), but ERROR-level log spam pollutes CloudWatch and obscures real failures.
## Root cause (best read)
The SDK calls `opentelemetry.context.attach(ctx)` to set span context, then `opentelemetry.context.detach(token)` to restore. Per PEP 567, every `asyncio.Task` (and every threadpool worker) gets its own copy of the contextvars state, so a token created in one execution context is not valid to `reset()` in another.
This shows up specifically when:
1. A boto3 sync call (e.g., `CreateEvent`) is invoked from inside an `async` method and ends up resuming on a different thread / task than the one that called `attach`.
2. A Strands session manager hook (e.g. `Created agent` event) attaches in one task and detaches after an `await` resumes on another task.
## Repro environment
- `bedrock-agentcore==1.6.2`
- `bedrock-agentcore-starter-toolkit==0.3.5`
- AgentCore-managed runtime (Lambda-style), `aws-opentelemetry-distro` enabled
- Strands `Agent` constructed with `AgentCoreMemorySessionManager`
- Concurrent module-builder agents launched from a `ThreadPoolExecutor`
## Expected
`detach()` should not raise (or the SDK should not call it across context boundaries). Either:
- Use `with use_span(...)` / `set_value` patterns that don't require explicit `detach`, or
- Capture the originating context and only detach if the current context still owns the token, or
- Wrap the detach in a try/except `ValueError` since a stale token can be safely ignored.
## Workaround consumers can apply
```python
from opentelemetry import context as _ctx
_orig_detach = _ctx.detach
def _safe_detach(token):
try:
_orig_detach(token)
except ValueError:
pass
_ctx.detach = _safe_detach
```
…installed before importing `bedrock_agentcore`. This is purely cosmetic and should live in the SDK instead.
## Suggested fix
In `bedrock_agentcore.memory.client` and `bedrock_agentcore.memory.integrations.strands.session_manager`, wrap each `context.detach(token)` call with a `try/except ValueError` (or refactor to use `opentelemetry.trace.use_span` so no manual token management is needed across `await` boundaries).
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu trong bedrock_agentcore/memory/client.py, quanh create_event và client.py:484, sau đó kiểm tra bedrock_agentcore/memory/integrations/strands/session_manager.py quanh dòng 406. Truy vết từng cặp attach/detach của OpenTelemetry qua các ranh giới async và thread, đồng thời tái hiện việc ghi log runtime đã được báo cáo. Hoàn tất khi các luồng memory và Strands vẫn giữ được tracing mà không có lỗi Failed to detach context.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- observability
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 68/100