microsoft / microsoft/foundry-local
Add AborSignal parameter to chat completions in ChatClient (JS)
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
When using Javascript, at the moment if you want to make chat completions to be cancellable, you need to use the OpenAI webservice to cancel your request. This works ok, but cancelling the completion streaming should also be doable using the internal API.
I'm not sure if the issue is also there for other languages, but here what the API would look like for Typescript :
class ChatClient {
....
completeStreamingChat(messages: any[], tools: any[], signal: AbortSignal ): AsyncIterable<any>;
}
This would allow user to stop models in user interface. Indeed it's common for users to see that the SLM has taken a wrong course of action, or is bugging (repeating same words again and again) and the ability to cancel these is necessary.
Edit:
Behaviour:
Cancelling chat completions should not cancel the session. users should be able to continue on the same session after chat a completion has been cancelled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.