`execute_tool` spans are siblings of `generate_content` instead of children — breaks observability span hierarchy
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 1k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 88
Description
## Summary
When an LLM agent performs a tool-calling flow (LLM → tool call → LLM summarize), the OpenTelemetry spans emitted by ADK have a **flat sibling structure** under `invoke_agent`, rather than a **nested hierarchy** that reflects the actual causal relationship. This makes it impossible for observability backends (Langfuse, Jaeger, etc.) to correctly display the logical execution order.
## Environment
- **ADK version**: `google.golang.org/adk v1.2.0`
- **Go version**: 1.24+
- **Observability backend**: Langfuse (via OTLP/HTTP), but this affects any OTel-compatible backend
## Current Behavior
When asking an LLM agent a question that requires a tool call (e.g., "What's the weather in Shanghai?"), the ADK emits the following span tree:
```
invoke_agent weather_time_agent (root span)
├── generate_content deepseek-v4-pro (Step 1: LLM returns FunctionCall)
├── execute_tool get_city_weather (Tool execution)
└── generate_content deepseek-v4-pro (Step 2: LLM summarizes tool result)
```
All three child spans (`generate_content`, `execute_tool`, `generate_content`) are **direct children of `invoke_agent`** at the same level. There is no span hierarchy that indicates `execute_tool` was triggered *because of* the first `generate_content` response.
In observability UIs like Langfuse, this flat structure causes the spans to be displayed in start-time order, but with no causal relationship visible. Worse, due to `BatchSpanProcessor` timing, the display order can appear as:
```
invoke_agent → generate_content → generate_content → execute_tool
```
which is misleading — it looks like two LLM calls happened before the tool was ever invoked.
## Expected Behavior
The span hierarchy should reflect the actual **causal/logical relationship** between steps within a single `runOneStep` cycle. Each `runOneStep` iteration should group its `generate_content` and resulting `execute_tool` spans together:
**Option A: Nested under `generate_content`**
```
invoke_agent weather_time_agent
├── generate_content deepseek-v4-pro (Step 1)
│ └── execute_tool get_city_weather
└── generate_content deepseek-v4-pro (Step 2: final response)
```
**Option B: Grouped under a `step` span**
```
invoke_agent weather_time_agent
├── step_1
│ ├── generate_content deepseek-v4-pro
│ └── execute_tool get_city_weather
└── step_2
└── generate_content deepseek-v4-pro
```
Either approach would give observability backends the information needed to render the correct causal flow.
## Impact
- **Observability accuracy**: Current flat hierarchy misleads developers when debugging agent flows
- **Cost tracking**: Backends like Langfuse cannot correctly attribute tool calls to the LLM invocation that triggered them
- **Multi-tool flows**: The problem compounds when an agent calls multiple tools across multiple LLM rounds — all `generate_content` and `execute_tool` spans appear as an unordered flat list
Contributor guide
Research direction
Start by tracing the runOneStep cycle and the instrumentation that emits the generate_content and execute_tool spans. Determine how the causal relationship should be represented, then verify that tool calls are nested or grouped with the generating LLM step and that multi-tool flows no longer appear as a flat sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100