apache / apache/apisix

help request: limit-req cannot limit access

Open
#11,288 12 comments 0 reactions 0 assignees View on GitHub
can't fix by us question
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

### Description

May I ask if the implementation principles of apisix limit req and nginx are the same? According to the official documentation, the leaky bucket algorithm is used. If the rate is 1000/s burst=0 nodelay, The number of access entries for nginx through access log grep 200 is less than the rate value, while apisix is much greater than the rate value

Referring to:
https://github.com/alibaba/tengine/issues/855
https://github.com/openresty/lua-resty-limit-traffic/blob/master/lib/resty/limit/req.md
https://groups.google.com/g/openresty/c/VY-LdQaEyDA/m/skf86NDHAAAJ?pli=1

route

```
{
"uri": "/rate",
"name": "rate-limit",
"priority": 100,
"host": "my-yace.com",
"plugins": {
"limit-req": {
"burst": 1,
"disable": false,
"nodelay": true,
"key": "server_addr",
"rate": 1000,
"rejected_code": 503
}
},
"upstream": {
"nodes": [
{
"host": "10.120.0.104",
"port": 80,
"weight": 1
}
],
"timeout": {
"connect": 6,
"send": 300,
"read": 300
},
"type": "roundrobin",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 300,
"requests": 1000,
"size": 1000
}
},
"status": 1
}
```

1、Using WRK testing

```
Running 1m test @ http://10.120.0.113:9080/rate

2 threads and 2000 connections

Thread calibration: mean lat.: 5041.922ms, rate sampling interval: 21381ms

Thread calibration: mean lat.: 5870.648ms, rate sampling interval: 21889ms

Thread Stats Avg Stdev Max +/- Stdev

Latency 33.93s 12.77s 0.96m 57.86%

Req/Sec 10.22k 38.16 10.26k 50.00%

1250705 requests in 1.00m, 313.49MB read

Socket errors: connect 0, read 3, write 628, timeout 2567

Non-2xx or 3xx responses: 421753

Requests/sec: 20844.28

Transfer/sec: 5.22MB
```

2、Confirm that the request has arrived at Apisix

```
\* About to connect() to 10.120.0.113 port 9080 (#0)

\* Trying 10.120.0.113...

\* Connected to 10.120.0.113 (10.120.0.113) port 9080 (#0)

\> GET /rate HTTP/1.1

\> User-Agent: curl/7.29.0

\> Accept: */*

\> Host: my-yace.com

\>

< HTTP/1.1 200 OK

< Content-Type: text/plain; charset=utf-8

< Content-Length: 0

< Connection: keep-alive

< Date: Fri, 24 May 2024 06:48:09 GMT

< Server: APISIX/3.2.2

<

\* Connection #0 to host 10.120.0.113 left intact
```

3 Number of grep log entries

```
grep 'GET /rate HTTP/1.1' access.log | grep 200 | grep '14:4' | awk '{print $4}' | sort | uniq -c

9334 [24/May/2024:14:40:00

9833 [24/May/2024:14:40:01

14500 [24/May/2024:14:40:02

14340 [24/May/2024:14:40:03

14648 [24/May/2024:14:40:04

14445 [24/May/2024:14:40:05

14849 [24/May/2024:14:40:06

13449 [24/May/2024:14:40:07

13938 [24/May/2024:14:40:08

13796 [24/May/2024:14:40:09

14089 [24/May/2024:14:40:10

14876 [24/May/2024:14:40:11

14499 [24/May/2024:14:40:12

12827 [24/May/2024:14:40:13

14015 [24/May/2024:14:40:14

14330 [24/May/2024:14:40:15

12880 [24/May/2024:14:40:16

13783 [24/May/2024:14:40:17

13856 [24/May/2024:14:40:18

14909 [24/May/2024:14:40:19

10330 [24/May/2024:14:40:20

10198 [24/May/2024:14:40:21

13245 [24/May/2024:14:40:22

13757 [24/May/2024:14:40:23

13998 [24/May/2024:14:40:24

10176 [24/May/2024:14:40:25

1478 [24/May/2024:14:40:26
```

4. Log fragments

4.1 access

```
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
```

4.2、 error

```
2024/05/24 14:40:26 [warn] 6913#6913: *83358 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *92234 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *80673 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6912#6912: *82365 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *86209 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *80674 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *83544 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *83847 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
```

### Environment

- APISIX version (run `apisix version`): 3.2.2
- Operating system (run `uname -a`): Linux nginx-gray02 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- nginx version: openresty/1.21.4.2
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1s 1 Nov 2022 (running with OpenSSL 1.1.1w 11 Sep 2023)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.2.2 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.4 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module

- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): {"id":"66356a45-bcc5-4a5a-9318-210342728ebd","boot_time":1716530994,"version":"3.2.2","etcd_version":"3.4.0","hostname":"nginx-gray02"}
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):

Contributor guide

Open the contributing guide

Research direction

Start with the limit-req plugin execution reported at plugin.lua:1100 and read the linked lua-resty-limit-traffic documentation and references. Reproduce the supplied route and wrk workload on APISIX 3.2.2, then compare access-log status counts with the 503 warnings. Done means the rate-limit behavior is explained and, if incorrect, a narrowly defined fix and regression test are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, nginx
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.