Introduce events in ChatClient streaming API

Open
#5,820 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
java, kotlin, spring

Research direction

Start by reading the existing StreamAdvisor and ChatClientResponse APIs and tracing the streaming call chain. Compare the proposed ChatClientEvent flow with current response handling; done means advisors can signal events through both streaming and blocking ChatClient APIs without using response context as a workaround.

Written by the indexing model from the issue text.

Description

status: waiting-for-triage

The advisor chain is a very powerful tool. However, I feel like there is a piece missing to the puzzle.

Whilst you can alter a ChatClientRequest or a ChatClientResponse in any way you'd like through an advisor, there is no clean way of signaling that something happened. The workaround I've found here is to inject data into the response's context and then looking for it after calling the ChatClient API, but to me that's a very hacky bandaid fix.

A practical example of this limitation is #5792, where the current API does not offer a clean way of propagating the tool responses to the user calling the ChatClient API.

To solve this, I'd like to propose introducing what I call a ChatClientEvent. This would be an interface used to signal anything that happens throughout the entire API call. Here's what it could look like:

interface ChatClientEvent {
    data class ModelGeneration(val generation: Generation) : ChatClientEvent
    data class ToolCalled(val id: String, val name: String, val responseData: String)  : ChatClientEvent
    //  any other events: InputValidationFailed, etc.
}

This interface could also be extended by users to define their own custom events.

The StreamAdvisor interface would be modified to return a Flux<ChatClientEvent>, like so:

public interface StreamAdvisor extends Advisor {
	Flux<ChatClientEvent> adviseStream(ChatClientRequest chatClientRequest, StreamAdvisorChain streamAdvisorChain);
}

Then, users could act on events just like they do with #chatClientResponse(), #chatResponse() or #content():

chatClient.prompt("Hello World!")
    .stream()
    .events()
    .doOnNext { event ->
        when(event) {
            is ModelGeneration -> logger.info("AI response: ${event.generation.output.text}, tool calls: ${event.generation.output.toolCalls}")
            is ToolCalled -> logger.info("Tool execution completed: id=${event.id}, tool=${event.name}, responseData=${event.responseData}")
        }
    }

The blocking API could also be updated to keep track of all events, adding an events field to the ChatClientResponse:

public record ChatClientResponse(@Nullable ChatResponse chatResponse, Map<String, @Nullable Object> context, List<ChatClientEvent> events)

I recognize that by adding the events field to ChatClientResponse, you could technically avoid introducing the #events() method to the streaming API and having to modify StreamAdvisor entirely. However - and this is more of a personal design choice, - I find it odd that the streaming API would return multiple ChatClientResponse objects, when conceptually it's all just one response that's being delivered in smaller chunks.

Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 10h
Merged PRs (30d)
5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from spring-projects/spring-ai

All issues in spring-projects/spring-ai

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.