abetlen / abetlen/llama-cpp-python
Feature request: add support for streaming tool use
Open
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.4k
- PR merge metrics
- PR metrics pending
Description
The combination `stream=True, tool_choice="auto"` raises an exception right now, which means that developers are stuck with one of two unfortunate choices:
1. Developing an application that streams the response but cannot use tools
2. Developing an LLM application that can use tools but cannot stream the response
Relevant discussion: https://github.com/abetlen/llama-cpp-python/discussions/1615
Contributor guide
Assessment
This issue has not been assessed yet.