spring-projects / spring-projects/spring-ai

Streaming Tool Calls problem in Observation

Open
#4,945 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Question

In flux streaming request, ChatClientObservationContext and ToolCallingObservationContext do not have a parent-child inheritance relationship, perhaps due to multithreading considerations. ChatClientObservationContext has an important indicator, spring.ai.chat.client.conversation.id, which could be passed to ToolCallingObservationContext to print when a tool call occurs. However, in the current Spring AI version 1.1.0, this doesn't seem to be possible yet. Is there any solution for this?

However, for call request, ChatClientObservationContext and ToolCallingObservationContext do have a parent-child inheritance relationship. This allows me to obtain the parent class ChatClientObservationContext, and thus the spring.ai.chat.client.conversation.id can be obtained.

public static ChatClientObservationContext findParentClientContext(Observation.ContextView ctx) {
        if (ctx == null) {
            return null;
        }
        try {
            ObservationView current = ctx.getParentObservation();
            while (current != null) {
                Observation.ContextView parentCtx = current.getContextView();
                if (parentCtx instanceof ChatClientObservationContext clientCtx) {
                    return clientCtx;
                }
                current = parentCtx.getParentObservation();
            }
        }catch (Throwable ignore) {
        }
        return null;
    }


public class ToolObservationHandler implements ObservationHandler<ToolCallingObservationContext> {
    @Override
    public void onStop(ToolCallingObservationContext context) {
           ChatClientObservationContext clientCtx = findParentClientContext(context);
                      // note: for flux streaming request, clientCtx will always be null
                      if (clientCtx != null) {
                          String conversationId = Optional.ofNullable(
                                          clientCtx.getHighCardinalityKeyValue(
                                                  ChatClientObservationDocumentation.HighCardinalityKeyNames.CHAT_CLIENT_CONVERSATION_ID.asString()))
                                  .map(KeyValue::getValue)
                                  .orElse("");
                      }
      }
}


// flux streaming request
 public Flux<String> flux(String message, Long chatId) {
        AtomicBoolean hasSentSeparator = new AtomicBoolean(false);

        return chatClient
                .prompt(promptService.getDefault())
                .user(message)
                .toolCallbacks(toolCallbackProvider.getToolCallbacks())
                .tools(localToolsService)
                .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, "abc-1"))
                .stream()
                .content().contextWrite(ctx -> ctx.put(ChatMemory.CONVERSATION_ID, "abc-1"));
    }

// `call` request
  public String chat(String message, Long chatId) {
        return chatClient
                .prompt(promptService.getDefault())
                .user(message)
                .toolCallbacks(toolCallbackProvider.getToolCallbacks())
                .tools(localToolsService)
                .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, "abc-1"))
                .call()
                .content();
    }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ChatClientObservationContext, ToolCallingObservationContext, and the ObservationHandler.onStop entry point, comparing the provided Flux streaming and call examples. Trace how parent observations and the conversation ID are handled in each path; done should be a defined, verified outcome for conversation-ID visibility during streaming tool calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.