openlit / openlit/openlit

Streamed tool calls produce spans without tool attributes (groq, ai21, together)

Open
#1,540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.8k
Forks
385
Avg merge
5d 11m
Merged PRs (30d)
34

Description

### Component
OpenLIT Python SDK

### What happened?

Streaming chat responses with tool calls produce spans with **no tool attributes at all** — no tool name, id, or arguments — for `groq`, `ai21`, and `together`. The tool-attribute logic in `common_chat_logic` exists and was hardened in #1452, but it only ever fires with data on the non-streaming path, because the streaming `process_chunk()` in each of these three instrumentors never populates `scope._tools`.

**Root cause (groq, reference file):**
- `sdk/python/src/openlit/instrumentation/groq/utils.py::process_chunk()` (~L265-300) reads `delta.content` and `delta.reasoning` from each chunk, but never reads `delta.tool_calls`.
- `scope._tools` is only ever assigned in the non-streaming `process_chat_response()` (~L682): `scope._tools = response_dict.get("choices", [{}])[0].get("message", {}).get("tool_calls")`.
- On the streaming path, `scope._tools` is initialized to `None` in `TracedSyncStream.__init__`/`TracedAsyncStream.__init__` (`groq.py`/`async_groq.py`) and never reassigned before `process_streaming_chat_response()` → `common_chat_logic()` runs. So the `if scope._tools:` block in `common_chat_logic` (~L425, hardened by #1452 to correctly handle `_tools` as a list instead of crashing) is dead code for streams — it just never triggers.
- Net effect: a streamed response with tool calls exports a span with token/cost/response-id attributes but zero tool attributes, indistinguishable from a plain text-only completion.

**Same shape in `ai21` and `together`:** both `sdk/python/src/openlit/instrumentation/ai21/utils.py::process_chunk()` and `sdk/python/src/openlit/instrumentation/together/utils.py::process_chunk()` only track `delta.content` and usage/finish_reason — no `tool_calls` handling. Both also only set `scope._tools` inside their respective (non-streaming) `process_chat_response()`. Same bug, same three-instrumentor blast radius as #1452/#1438.

**`reka` is not affected** — `reka.py`/`async_reka.py` only wire up `process_chat_response` (no `process_chunk`/streaming wrapper exists in that instrumentor at all), so there's no streaming path to lose tool attributes on.

**`mistral` already does this correctly** and is a good reference for the fix: `sdk/python/src/openlit/instrumentation/mistral/utils.py::process_chunk()` accumulates `delta.tool_calls` into `scope._tools` chunk-by-chunk (matching by `index`, building up `id`/`function.name`/`function.arguments` incrementally, appending streamed argument fragments) before the final chunk's usage triggers `process_streaming_chat_response`.

**Related context:** this surfaced while reviewing #1516, which fixes Groq streaming spans getting lost entirely when the caller calls `stream.close()` early — described in that PR as "the canonical agentic tool-calling pattern." #1516 rescues exactly the spans this issue affects: once #1516 lands, streamed agentic tool-call spans will reliably export, but still with no tool attributes on them.

### Steps to reproduce

```python
with client.chat.completions.create(
model="llama-3.1-8b-instant",
messages=[...],
tools=[...],
stream=True,
) as stream:
for chunk in stream:
pass # or break/close() per #1516
```

Inspect the exported span: token/cost/response-id attributes are present, but no `gen_ai.content.completion.tool_calls` / tool name / tool id / tool arguments attributes, even though the model called a tool.

### Expected behavior

A streamed response containing tool calls should export the same tool attributes (name, id, arguments) as the equivalent non-streaming call.

### Suggested fix

Port the accumulation pattern already used in `mistral/utils.py::process_chunk()` into `groq/utils.py`, `ai21/utils.py`, and `together/utils.py`: read `delta.tool_calls` per chunk, index-match against a growing `scope._tools` list, and append argument fragments across chunks. No changes needed to `common_chat_logic` itself — that block already handles `scope._tools` correctly as of #1452, it just needs to actually receive data on the streaming path.

### Relevant log output

_No response_

### Twitter / LinkedIn details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with sdk/python/src/openlit/instrumentation/mistral/utils.py::process_chunk() to understand the existing accumulation pattern. Compare it with process_chunk() in groq/utils.py, ai21/utils.py, and together/utils.py, then verify that streamed tool calls populate scope._tools and export the same tool name, id, and arguments as non-streaming calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.