aws / aws/bedrock-agentcore-starter-toolkit
[BUG] CloudWatchAgentSpanCollector fails to collect openinference.instrumentation.langchain spans for LangGraph agents deployed via CDK
- Dominant language
- Python
- Stars
- 508
- Forks
- 155
- Avg merge
- 8h 50m
- Merged PRs (30d)
- 4
Description
## Description
When using `OnDemandEvaluationDatasetRunner` with `CloudWatchAgentSpanCollector` to evaluate a LangGraph agent deployed on AgentCore Runtime via CDK (not AgentCore CLI), the built-in evaluators fail with:
```
ValidationException: Provided input has no spans with supported scope.
Currently supported scopes are ['strands.telemetry.tracer',
'opentelemetry.instrumentation.langchain', 'openinference.instrumentation.langchain']
```
## Environment
- bedrock-agentcore SDK: 1.7.0
- openinference-instrumentation-langchain: 0.1.63
- openinference-instrumentation-bedrock: (latest)
- opentelemetry-instrumentation-langchain: 0.60.0
- aws-opentelemetry-distro: (latest)
- langgraph: 1.1.10
- langchain-aws: 1.4.5 (ChatBedrockConverse)
- Python: 3.12 (ARM64 on AgentCore Runtime)
- Region: us-east-1
- Deployment method: AWS CDK (not AgentCore CLI)
- Transaction Search: enabled at 100% sampling
## What works
- Spans with `openinference.instrumentation.langchain` scope ARE present in `aws/spans` log group (verified via CloudWatch Insights manual query — 35 results found)
- Spans with `openinference.instrumentation.bedrock` scope also present
- `session.id` attribute correctly set on spans in `aws/spans`
- Simple scenarios (greeting — no tool calls, 1 LLM call) evaluate successfully: GoalSuccessRate=1.0, Correctness=1.0, Helpfulness=0.83
- Agent invocations succeed for all scenarios
## What fails
- All scenarios involving tool calls (MCP Gateway → Lambda) fail with "no spans with supported scope"
- The `CloudWatchAgentSpanCollector._fetch_spans()` queries both `aws/spans` and the Runtime log group, but the collected spans passed to the Evaluate API don't include the `openinference.instrumentation.langchain` scoped ones
## Steps to reproduce
1. Deploy a LangGraph agent on AgentCore Runtime via CDK with `ChatBedrockConverse` for LLM calls and `langchain-mcp-adapters` for tool calls
2. Add `openinference-instrumentation-langchain`, `openinference-instrumentation-bedrock`, and `aws-opentelemetry-distro` to the agent bundle
3. Set OTEL env vars: `AGENT_OBSERVABILITY_ENABLED=true`, `OTEL_PYTHON_DISTRO=aws_distro`, `OTEL_PYTHON_CONFIGURATOR=aws_configurator`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`
4. Set entrypoint to `["opentelemetry-instrument", "main.py"]`
5. Explicitly initialize instrumentors in main.py:
```python
from openinference.instrumentation.langchain import LangChainInstrumentor
LangChainInstrumentor().instrument()
from openinference.instrumentation.bedrock import BedrockInstrumentor
BedrockInstrumentor().instrument()
```
6. Enable Transaction Search at 100% sampling
7. Run `OnDemandEvaluationDatasetRunner` with `CloudWatchAgentSpanCollector`
## Evidence
### Spans exist in aws/spans with correct scope
CloudWatch Insights query on `aws/spans`:
```
fields @timestamp, @message
| filter @message like "list-vendors-3e2c9f9f"
| limit 5
```
Returns 5 results with spans like:
```json
{
"scope": {
"name": "openinference.instrumentation.langchain",
"version": "0.1.63"
},
"name": "nexusflow-dynamodb-query___list_vendors",
"attributes": {
"openinference.span.kind": "TOOL",
"session.id": "list-vendors-3e2c9f9f-63a2-4191-9f16-ea1d711aaa31",
"tool.name": "nexusflow-dynamodb-query___list_vendors"
}
}
```
### SDK's own query format also finds spans
Running the exact query from `CloudWatchAgentSpanCollector._fetch_spans()`:
```
fields @timestamp, @message
| filter attributes.session.id = "list-vendors-dece0b51-ac1d-461f-b8e0-6ab724f60bd6"
| filter ispresent(scope.name)
| filter ispresent(traceId)
| filter ispresent(spanId)
| sort @timestamp asc
```
Returns 35 results.
### But the SDK returns 0 spans
Despite the manual query returning 35 results, `CloudWatchAgentSpanCollector.collect()` returns 0 spans for the same session ID, causing all evaluators to fail.
## Suspected root cause
The `CloudWatchAgentSpanCollector._helper.query_log_group()` method may not be correctly parsing the CloudWatch Insights query results from `aws/spans`, or there is a time window mismatch between when the SDK queries and when the spans become available for Insights queries.
Additionally, multi-step LangGraph scenarios create multiple trace contexts (due to `asyncio.run()` in tool invocations breaking trace propagation), which may cause the span collector to miss spans that belong to the same session but have different trace IDs.
## Workaround
Simple scenarios without tool calls (e.g., greeting with 1 LLM call) evaluate successfully, suggesting the issue is specific to multi-step graph execution with MCP tool calls.
Contributor guide
Research direction
Start with CloudWatchAgentSpanCollector.collect() and _fetch_spans(), then inspect _helper.query_log_group() and the OnDemandEvaluationDatasetRunner path. Compare the SDK's returned spans with the 35 results from the documented CloudWatch Insights query, including multi-trace tool-call scenarios. Done means collected spans retain supported scopes and tool-call evaluations no longer fail with "no spans with supported scope".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100