deepset-ai / deepset-ai/haystack
Streamlit Stream Callback in Haystack 2.x
Open
Nobody has claimed this yet.
P2
topic:streaming
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Describe the solution you'd like
I would like to see streaming response in real time on streamlit when using OpenAIChatGenerator
Describe alternatives you've considered
I have tried to the following streaming_callback function. (I have only put related code fields. I can show full code if requested)
import streamlit as st
# Method 1: Nothing seen in the UI
def st_streaming_callback(chunk: StreamingChunk):
yield chunk.content
# Method 2: Instead of typewriter format, each chunk seen as one line output.
def st_streaming_callback(chunk: StreamingChunk):
st.write(chunk.content)
model = OpenAIChatGenerator(
model="gpt-4-turbo-preview",
streaming_callback=streaming_callback,
generation_kwargs=None,
)
response_dict = st.write_stream(
llm_chat.run(
messages=[
ChatMessage(
role=message.role, content=message.content, name=None
)
for message in st.session_state.messages
]
)
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.