google / google/adk-go

`execute_tool` spans are siblings of `generate_content` instead of children — breaks observability span hierarchy

Open
#857 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
8.8k
Forks
1k
Avg merge
3d 18h
Merged PRs (30d)
88

Description

## Summary

When an LLM agent performs a tool-calling flow (LLM → tool call → LLM summarize), the OpenTelemetry spans emitted by ADK have a **flat sibling structure** under `invoke_agent`, rather than a **nested hierarchy** that reflects the actual causal relationship. This makes it impossible for observability backends (Langfuse, Jaeger, etc.) to correctly display the logical execution order.

## Environment

- **ADK version**: `google.golang.org/adk v1.2.0`
- **Go version**: 1.24+
- **Observability backend**: Langfuse (via OTLP/HTTP), but this affects any OTel-compatible backend

## Current Behavior

When asking an LLM agent a question that requires a tool call (e.g., "What's the weather in Shanghai?"), the ADK emits the following span tree:

```
invoke_agent weather_time_agent (root span)
├── generate_content deepseek-v4-pro (Step 1: LLM returns FunctionCall)
├── execute_tool get_city_weather (Tool execution)
└── generate_content deepseek-v4-pro (Step 2: LLM summarizes tool result)
```

All three child spans (`generate_content`, `execute_tool`, `generate_content`) are **direct children of `invoke_agent`** at the same level. There is no span hierarchy that indicates `execute_tool` was triggered *because of* the first `generate_content` response.

In observability UIs like Langfuse, this flat structure causes the spans to be displayed in start-time order, but with no causal relationship visible. Worse, due to `BatchSpanProcessor` timing, the display order can appear as:

```
invoke_agent → generate_content → generate_content → execute_tool
```

which is misleading — it looks like two LLM calls happened before the tool was ever invoked.

## Expected Behavior

The span hierarchy should reflect the actual **causal/logical relationship** between steps within a single `runOneStep` cycle. Each `runOneStep` iteration should group its `generate_content` and resulting `execute_tool` spans together:

**Option A: Nested under `generate_content`**
```
invoke_agent weather_time_agent
├── generate_content deepseek-v4-pro (Step 1)
│ └── execute_tool get_city_weather
└── generate_content deepseek-v4-pro (Step 2: final response)
```

**Option B: Grouped under a `step` span**
```
invoke_agent weather_time_agent
├── step_1
│ ├── generate_content deepseek-v4-pro
│ └── execute_tool get_city_weather
└── step_2
└── generate_content deepseek-v4-pro
```

Either approach would give observability backends the information needed to render the correct causal flow.

## Impact

- **Observability accuracy**: Current flat hierarchy misleads developers when debugging agent flows
- **Cost tracking**: Backends like Langfuse cannot correctly attribute tool calls to the LLM invocation that triggered them
- **Multi-tool flows**: The problem compounds when an agent calls multiple tools across multiple LLM rounds — all `generate_content` and `execute_tool` spans appear as an unordered flat list

Contributor guide

Open the contributing guide

Research direction

Start by tracing the runOneStep cycle and the instrumentation that emits the generate_content and execute_tool spans. Determine how the causal relationship should be represented, then verify that tool calls are nested or grouped with the generating LLM step and that multi-tool flows no longer appear as a flat sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.