apache / apache/apisix

feat: more precise control over the timeout for requests to the LLM service

Open
#12,072 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

### Description

Currently, AI-related plugins only have a timeout parameter. After reviewing the code, I found that timeout is the total of connect, read, and send timeout durations, making it impossible to precisely control the timeout for requests to the LLM service. This needs to be improved.

Currently https://github.com/apache/apisix/blob/master/apisix/plugins/ai-drivers/openai-base.lua#L70
```lua
...
local http = require("resty.http")
...

httpc:set_timeout(conf.timeout)
```

Expected:
```lua
...
local http = require("resty.http")
...

httpc:set_timeouts(connect_timeout, send_timeout, read_timeout)
```

Contributor guide

Open the contributing guide

Research direction

Start with apisix/plugins/ai-drivers/openai-base.lua around the current httpc:set_timeout(conf.timeout) call, then trace how the AI plugin timeout configuration reaches resty.http. Check the existing AI plugin coverage before making the change. Done means requests can use separate connect, send, and read timeout values as described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
ai, api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.