spring-projects / spring-projects/spring-ai
Optimized aggregation advisors for streaming scenarios
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 5
Description
In the latest round of architectural changes of Advisors in #1422 there now are two types of advisors:
CallAroundAdvisorStreamAroundAdvisor
In the case of the non-streaming one, it's easy to take some actions based on the entire response. However, the streaming manipulates an entire stream. A next advisor in the chain can also manipulate the entire stream. If a stream advisor in the middle is acting upon each chunk of the response, all should be fine. However, if the advisor is only interested in the entire aggregation it would modify the stream in a way that aggregates everything in a side channel, e.g. using org.springframework.ai.chat.model.MessageAggregator class. If multiple advisors perform the same type of aggregation it is inefficient in terms of both time and memory.
Having that, I propose a new interface, StreamAggregationAdvisor. Instances of this type would be fed with an aggregation of the original stream of chunks coming back from the model on their way into the application before any other advisors have a chance to manipulate the stream. The aggregation would then be performed once and could deal with the unaltered view of the exchange. The way to implement this behaviour would be based on utilizing the innermost StreamAroundAdvisor that is created in the DefaultChatClient.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the advisor architectural changes in #1422, then inspect the innermost StreamAroundAdvisor created in DefaultChatClient and the existing MessageAggregator. Done means StreamAggregationAdvisor instances receive one aggregation of the unaltered model stream before other advisors can manipulate it, avoiding repeated aggregation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100