Support for stream: false via extra-openai-models.yaml
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 998
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Description
It seems that register_model() in openai_models.py doesn't currently expect a stream variable to be in the .yaml file, and can_stream gets set to True by default.
Many organizations use an internal OpenAI-compatible API proxy or gateway to access OpenAI and control the keys. For this, extra-openai-models.yaml is the easy way to make proxies just work. However, currently there doesn't seem to be a mechanism to pass the equivalent of --no-stream via this .yaml. All OpenAI-compatible gateways are expected to be able to stream.
This makes --no-stream an obligatory option at CLI runtime with quick PoC API gateways, that for example implement key sharing or team-level usage limits for the org.
I've tried writing a streaming API proxy myself, and turns out streaming is not that trivial to implement. Non-streaming proxies are easy, so I suspect many teams in their hurry start with that. There are some streaming-capable proxy projects in Github, but I think it'd be logical if stream: false or can_stream: false via extra-openai-models.yaml was passed downstream to execute().
Or is there a better, already-supported way to do this? Such as model default options saved by the user? A bit like aliases are. I do admit that "how to call a model" and "what are its default options" isn't quite the same level question to define in the .yaml, design-wise.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with register_model() in openai_models.py and the extra-openai-models.yaml format, then trace how can_stream and execute() handle model options. Compare this path with the existing --no-stream CLI behavior and resolve whether stream or can_stream is the intended configuration name. Done means a YAML-configured model can disable streaming without requiring the CLI flag.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100