awslabs / awslabs/llmeter

OpenAI timeouts are broken, and Runner doesn't enforce them consistently

Open
#78 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

Per https://github.com/openai/openai-python/issues/2599, timeout settings are not actually being respected in upstream OpenAI Python SDK - and therefore also not in our OpenAI Endpoints.

This seems mostly like something that should be fixed upstream rather than worked around in LLMeter - but then I noticed the Runner does make [some attempt](https://github.com/awslabs/llmeter/blob/406f5cc62eb1b4509bbf04e9fc64b661730c108b/llmeter/runner.py#L482) to enforce that `_invoke_n_no_wait` is abandoned after N requests * `Runner.timeout` (for count-bound runs only)... And the [Runner docstring](https://github.com/awslabs/llmeter/blob/406f5cc62eb1b4509bbf04e9fc64b661730c108b/llmeter/runner.py#L778) does claim that the timeout setting is enforced (without disclaiming different run types). I'd note that even for count-bound runs, enforcing total timeout is not the same as enforcing request-level timeout.

We should try to clarify the behaviour here and improve its consistency if practical. If Runner-level enforcement is possible, it could be a nice guard against inconsistencies or errors in Endpoint implementations... But `_invoke_n_no_wait` is currently just a synchronous while loop of direct `self._endpoint.invoke(p)` so i don't think it'll be easy to add without another layer of asyncio/thread delegation.

Contributor guide

Open the contributing guide

Research direction

Start in llmeter/runner.py at _invoke_n_no_wait and the Runner docstring around the referenced lines, then review the upstream OpenAI timeout issue. Clarify the behavior for count-bound and other run types, and determine whether Runner-level enforcement can be made consistent without the proposed asyncio or thread delegation. Done means the documented timeout guarantees match the implemented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.