aws / aws/bedrock-agentcore-starter-toolkit

[BUG] CloudWatchAgentSpanCollector fails to collect openinference.instrumentation.langchain spans for LangGraph agents deployed via CDK

Open
#509 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
508
Forks
155
Avg merge
8h 50m
Merged PRs (30d)
4

Description

## Description

When using `OnDemandEvaluationDatasetRunner` with `CloudWatchAgentSpanCollector` to evaluate a LangGraph agent deployed on AgentCore Runtime via CDK (not AgentCore CLI), the built-in evaluators fail with:

```
ValidationException: Provided input has no spans with supported scope.
Currently supported scopes are ['strands.telemetry.tracer',
'opentelemetry.instrumentation.langchain', 'openinference.instrumentation.langchain']
```

## Environment

- bedrock-agentcore SDK: 1.7.0
- openinference-instrumentation-langchain: 0.1.63
- openinference-instrumentation-bedrock: (latest)
- opentelemetry-instrumentation-langchain: 0.60.0
- aws-opentelemetry-distro: (latest)
- langgraph: 1.1.10
- langchain-aws: 1.4.5 (ChatBedrockConverse)
- Python: 3.12 (ARM64 on AgentCore Runtime)
- Region: us-east-1
- Deployment method: AWS CDK (not AgentCore CLI)
- Transaction Search: enabled at 100% sampling

## What works

- Spans with `openinference.instrumentation.langchain` scope ARE present in `aws/spans` log group (verified via CloudWatch Insights manual query — 35 results found)
- Spans with `openinference.instrumentation.bedrock` scope also present
- `session.id` attribute correctly set on spans in `aws/spans`
- Simple scenarios (greeting — no tool calls, 1 LLM call) evaluate successfully: GoalSuccessRate=1.0, Correctness=1.0, Helpfulness=0.83
- Agent invocations succeed for all scenarios

## What fails

- All scenarios involving tool calls (MCP Gateway → Lambda) fail with "no spans with supported scope"
- The `CloudWatchAgentSpanCollector._fetch_spans()` queries both `aws/spans` and the Runtime log group, but the collected spans passed to the Evaluate API don't include the `openinference.instrumentation.langchain` scoped ones

## Steps to reproduce

1. Deploy a LangGraph agent on AgentCore Runtime via CDK with `ChatBedrockConverse` for LLM calls and `langchain-mcp-adapters` for tool calls
2. Add `openinference-instrumentation-langchain`, `openinference-instrumentation-bedrock`, and `aws-opentelemetry-distro` to the agent bundle
3. Set OTEL env vars: `AGENT_OBSERVABILITY_ENABLED=true`, `OTEL_PYTHON_DISTRO=aws_distro`, `OTEL_PYTHON_CONFIGURATOR=aws_configurator`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`
4. Set entrypoint to `["opentelemetry-instrument", "main.py"]`
5. Explicitly initialize instrumentors in main.py:
```python
from openinference.instrumentation.langchain import LangChainInstrumentor
LangChainInstrumentor().instrument()
from openinference.instrumentation.bedrock import BedrockInstrumentor
BedrockInstrumentor().instrument()
```
6. Enable Transaction Search at 100% sampling
7. Run `OnDemandEvaluationDatasetRunner` with `CloudWatchAgentSpanCollector`

## Evidence

### Spans exist in aws/spans with correct scope

CloudWatch Insights query on `aws/spans`:
```
fields @timestamp, @message
| filter @message like "list-vendors-3e2c9f9f"
| limit 5
```
Returns 5 results with spans like:
```json
{
"scope": {
"name": "openinference.instrumentation.langchain",
"version": "0.1.63"
},
"name": "nexusflow-dynamodb-query___list_vendors",
"attributes": {
"openinference.span.kind": "TOOL",
"session.id": "list-vendors-3e2c9f9f-63a2-4191-9f16-ea1d711aaa31",
"tool.name": "nexusflow-dynamodb-query___list_vendors"
}
}
```

### SDK's own query format also finds spans

Running the exact query from `CloudWatchAgentSpanCollector._fetch_spans()`:
```
fields @timestamp, @message
| filter attributes.session.id = "list-vendors-dece0b51-ac1d-461f-b8e0-6ab724f60bd6"
| filter ispresent(scope.name)
| filter ispresent(traceId)
| filter ispresent(spanId)
| sort @timestamp asc
```
Returns 35 results.

### But the SDK returns 0 spans

Despite the manual query returning 35 results, `CloudWatchAgentSpanCollector.collect()` returns 0 spans for the same session ID, causing all evaluators to fail.

## Suspected root cause

The `CloudWatchAgentSpanCollector._helper.query_log_group()` method may not be correctly parsing the CloudWatch Insights query results from `aws/spans`, or there is a time window mismatch between when the SDK queries and when the spans become available for Insights queries.

Additionally, multi-step LangGraph scenarios create multiple trace contexts (due to `asyncio.run()` in tool invocations breaking trace propagation), which may cause the span collector to miss spans that belong to the same session but have different trace IDs.

## Workaround

Simple scenarios without tool calls (e.g., greeting with 1 LLM call) evaluate successfully, suggesting the issue is specific to multi-step graph execution with MCP tool calls.

Contributor guide

Open the contributing guide

Research direction

Start with CloudWatchAgentSpanCollector.collect() and _fetch_spans(), then inspect _helper.query_log_group() and the OnDemandEvaluationDatasetRunner path. Compare the SDK's returned spans with the 35 results from the documented CloudWatch Insights query, including multi-trace tool-call scenarios. Done means collected spans retain supported scopes and tool-call evaluations no longer fail with "no spans with supported scope".

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.