The invoker pod was cleaned up for some reason and the function call timed out.
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
apisix calls openwhisk, when the invoker pod is cleaned up for some reason, this will cause apisix to return 503.
POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&result=true&timeout=8000 HTTP/1.1 499
## Environment details:
* Kubernetes 1.26
## Steps to reproduce the issue:
1. kubectl get po -n openwhisk -owide
2. kubectl delete po -n openwhisk wskowdev-invoker-00-65-guest-getuser
```
# kubectl get po -n openwhisk
NAME READY STATUS RESTARTS AGE
owdev-alarmprovider-67b79fcf46-m7jl8 1/1 Running 0 62d
owdev-apigateway-6fc74d7fcd-4442p 1/1 Running 0 62d
owdev-controller-0 1/1 Running 0 2d23h
owdev-couchdb-84b8749bbf-dcstw 1/1 Running 0 62d
owdev-gen-certs-zcqnh 0/1 Completed 0 62d
owdev-init-couchdb-nfhv7 0/1 Completed 0 62d
owdev-invoker-0 1/1 Running 0 2d22h
owdev-kafka-0 1/1 Running 0 62d
owdev-kafkaprovider-b4ddfd888-bwk66 1/1 Running 0 62d
owdev-nginx-844c58dd89-ghdmp 1/1 Running 0 62d
owdev-redis-57b7889b49-g7pf6 1/1 Running 0 62d
owdev-wskadmin 1/1 Running 0 62d
owdev-zookeeper-0 1/1 Running 0 62d
wskowdev-invoker-00-18-prewarm-go115 1/1 Running 0 2d22h
wskowdev-invoker-00-19-prewarm-go115 1/1 Running 0 2d22h
wskowdev-invoker-00-65-guest-getuser 1/1 Running 0 21s
```
3. Call the function
## Provide the expected results and outputs:
```
Normal return function call results
```
## Provide the actual results and outputs:
```
root@owdev-nginx-844c58dd89-ghdmp:/logs# tail -f *
==> nginx_access.log <==
172.22.17.143 - 23bc46b1-71f6-4ed5-8c54-816aa4f8c502 [09/Sep/2024:06:38:36 +0000] [#tid_806bbecd8407144cde4ba08e42459828] POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&result=true&timeout=8000 HTTP/1.1 499 0 - lua-resty-http/0.16.1 (Lua) ngx_lua/10025 10.103.10.24:8080
172.22.49.138 - 23bc46b1-71f6-4ed5-8c54-816aa4f8c502 [09/Sep/2024:06:39:08 +0000] [#tid_a349ba08dedc07338ffdcff80153ede4] POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=8000&result=true HTTP/1.1 499 0 - lua-resty-http/0.16.1 (Lua) ngx_lua/10025 10.103.10.24:8080
==> nginx_error.log <==
2024/08/26 10:08:56 [error] 31#31: *14464 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.49.138, server: ~^(?[0-9a-zA-Z-]+)\.$, request: "POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=60000&result=true HTTP/1.1", upstream: "http://10.103.10.24:8080/api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=60000&result=true", host: "10.103.184.89"
```
Contributor guide
Assessment
This issue has not been assessed yet.