OpenAI timeouts are broken, and Runner doesn't enforce them consistently
- Dominant language
- Python
- Stars
- 41
- Forks
- 5
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Per https://github.com/openai/openai-python/issues/2599, timeout settings are not actually being respected in upstream OpenAI Python SDK - and therefore also not in our OpenAI Endpoints.
This seems mostly like something that should be fixed upstream rather than worked around in LLMeter - but then I noticed the Runner does make [some attempt](https://github.com/awslabs/llmeter/blob/406f5cc62eb1b4509bbf04e9fc64b661730c108b/llmeter/runner.py#L482) to enforce that `_invoke_n_no_wait` is abandoned after N requests * `Runner.timeout` (for count-bound runs only)... And the [Runner docstring](https://github.com/awslabs/llmeter/blob/406f5cc62eb1b4509bbf04e9fc64b661730c108b/llmeter/runner.py#L778) does claim that the timeout setting is enforced (without disclaiming different run types). I'd note that even for count-bound runs, enforcing total timeout is not the same as enforcing request-level timeout.
We should try to clarify the behaviour here and improve its consistency if practical. If Runner-level enforcement is possible, it could be a nice guard against inconsistencies or errors in Endpoint implementations... But `_invoke_n_no_wait` is currently just a synchronous while loop of direct `self._endpoint.invoke(p)` so i don't think it'll be easy to add without another layer of asyncio/thread delegation.
Contributor guide
Research direction
Start in llmeter/runner.py at _invoke_n_no_wait and the Runner docstring around the referenced lines, then review the upstream OpenAI timeout issue. Clarify the behavior for count-bound and other run types, and determine whether Runner-level enforcement can be made consistent without the proposed asyncio or thread delegation. Done means the documented timeout guarantees match the implemented behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100