aws / aws/bedrock-agentcore-sdk-python

OTEL context token detached across asyncio/thread boundary in memory client + Strands session_manager

Offen
#456 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
761
Forks
147
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
7

Beschreibung

## Summary

When using `bedrock-agentcore` (1.6.2) with the Strands integration inside an AgentCore runtime, the OpenTelemetry runtime logs a steady stream of `Failed to detach context` errors:

```
ERROR [opentelemetry.context] [__init__.py:157] - Failed to detach context
Traceback (most recent call last):
File "/var/task/opentelemetry/context/__init__.py", line 155, in detach
_RUNTIME_CONTEXT.detach(token)
File "/var/task/opentelemetry/context/contextvars_context.py", line 53, in detach
self._current_context.reset(token)
ValueError: at 0x...> was created in a different Context
```

The errors are logged from spans owned by:

- `bedrock_agentcore.memory.client` (around `create_event` / `client.py:484`)
- `bedrock_agentcore.memory.integrations.strands.session_manager` (`session_manager.py:406`, around `Created agent: default in session: …`)

Functionally tracing still works (events are created, agents start), but ERROR-level log spam pollutes CloudWatch and obscures real failures.

## Root cause (best read)

The SDK calls `opentelemetry.context.attach(ctx)` to set span context, then `opentelemetry.context.detach(token)` to restore. Per PEP 567, every `asyncio.Task` (and every threadpool worker) gets its own copy of the contextvars state, so a token created in one execution context is not valid to `reset()` in another.

This shows up specifically when:

1. A boto3 sync call (e.g., `CreateEvent`) is invoked from inside an `async` method and ends up resuming on a different thread / task than the one that called `attach`.
2. A Strands session manager hook (e.g. `Created agent` event) attaches in one task and detaches after an `await` resumes on another task.

## Repro environment

- `bedrock-agentcore==1.6.2`
- `bedrock-agentcore-starter-toolkit==0.3.5`
- AgentCore-managed runtime (Lambda-style), `aws-opentelemetry-distro` enabled
- Strands `Agent` constructed with `AgentCoreMemorySessionManager`
- Concurrent module-builder agents launched from a `ThreadPoolExecutor`

## Expected

`detach()` should not raise (or the SDK should not call it across context boundaries). Either:

- Use `with use_span(...)` / `set_value` patterns that don't require explicit `detach`, or
- Capture the originating context and only detach if the current context still owns the token, or
- Wrap the detach in a try/except `ValueError` since a stale token can be safely ignored.

## Workaround consumers can apply

```python
from opentelemetry import context as _ctx
_orig_detach = _ctx.detach
def _safe_detach(token):
try:
_orig_detach(token)
except ValueError:
pass
_ctx.detach = _safe_detach
```

…installed before importing `bedrock_agentcore`. This is purely cosmetic and should live in the SDK instead.

## Suggested fix

In `bedrock_agentcore.memory.client` and `bedrock_agentcore.memory.integrations.strands.session_manager`, wrap each `context.detach(token)` call with a `try/except ValueError` (or refactor to use `opentelemetry.trace.use_span` so no manual token management is needed across `await` boundaries).

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne in bedrock_agentcore/memory/client.py bei create_event und client.py:484 und untersuche anschließend bedrock_agentcore/memory/integrations/strands/session_manager.py etwa bei Zeile 406. Verfolge jedes OpenTelemetry-Attach/Detach-Paar über Async- und Thread-Grenzen hinweg und reproduziere die gemeldete Laufzeitprotokollierung. Die Arbeit ist abgeschlossen, wenn die Memory- und Strands-Abläufe das Tracing ohne Failed to detach context-Fehler beibehalten.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
observability
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
68/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.