deepset-ai / deepset-ai/haystack-core-integrations

bug(langfuse): content tracing fails Agent runs when a tool returns None, a string, or an image

Open
#3,961 0 comments 0 reactions 1 assignee View on GitHub

@sjrl is already working on this.

Since Sep 15, 2026.

Dominant language
Python
Stars
203
Forks
332
Avg merge
2d 4h
Merged PRs (30d)
80

Description

Describe the bug

With HAYSTACK_CONTENT_TRACING_ENABLED=true and LangfuseConnector, LangfuseSpan.set_content_tag can raise inside the traced run, so the whole Agent or Pipeline run fails. With tracing off, the same run succeeds.

Since Haystack 3.0 (deepset-ai/haystack#11892), the Agent creates one span per tool call and passes the raw tool arguments and result to set_content_tag. The Langfuse tracer assumes that every *.input value containing "messages" is a dict of ChatMessages, and that every *.output value containing "replies" is a dict. Values that don't fit this break it:

Value traced Error
Tool returns None, an int, a float or a bool TypeError: argument of type 'NoneType' is not iterable (or 'int' etc.)
Tool returns a string containing "replies", e.g. "No replies found" AttributeError: 'str' object has no attribute 'get'
Tool returns an image (outputs_to_string={"raw_result": True}) ValueError: Unsupported tool result from ChatMessage.to_openai_dict_format, raised on the next LLM step's input
A component input named messages that isn't list[ChatMessage], e.g. list[str] AttributeError: 'str' object has no attribute 'to_openai_dict_format'

To Reproduce

This needs only haystack-ai and langfuse-haystack. Any Langfuse keys work: the error doesn't depend on spans actually being exported. A scripted chat generator stands in for a real one so that no LLM API key is needed. OpenAIChatGenerator behaves the same way.

import os

os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
os.environ.setdefault("LANGFUSE_PUBLIC_KEY", "pk-lf-...")
os.environ.setdefault("LANGFUSE_SECRET_KEY", "sk-lf-...")

from haystack import component
from haystack.components.agents import Agent
from haystack.dataclasses import ChatMessage, ToolCall
from haystack.tools import tool

from haystack_integrations.components.connectors.langfuse import LangfuseConnector


@component
class ScriptedChatGenerator:
    """Calls the tool once, then answers."""

    def __init__(self):
        self.calls = 0

    @component.output_types(replies=list[ChatMessage])
    def run(self, messages: list[ChatMessage], tools=None, **kwargs):
        self.calls += 1
        if self.calls == 1:
            return {"replies": [ChatMessage.from_assistant(tool_calls=[ToolCall(tool_name="notify", arguments={"to": "bob"}, id="call_1")])]}
        return {"replies": [ChatMessage.from_assistant("Done")]}


@tool
def notify(to: str) -> None:
    """Send a notification to a user."""


LangfuseConnector("repro").warm_up()
agent = Agent(chat_generator=ScriptedChatGenerator(), tools=[notify])
print(agent.run(messages=[ChatMessage.from_user("Tell Bob")])["last_message"].text)
# -> TypeError: argument of type 'NoneType' is not iterable

Expected behavior

Tracing never makes a run fail. Values that aren't chat messages are recorded with tracing_utils.coerce_tag_value, like other tags.

Describe your environment (please complete the following information):

  • OS: macOS
  • Haystack version: haystack-ai 3.1.1
  • Integration version: langfuse-haystack 6.0.0 (also main); langfuse 4.15.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.