apache / apache/apisix

help request: request etcd endpoint timeout error

Open
#12,832 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

### Current Behavior

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint "http://apisix-etcd:2379/v3/kv/put" error, timeout
trying to initialize the data of etcd
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=1
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=1
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=2

In this Kubernetes cluster, APISIX 3.14.1 is deployed via Helm, but one of the APISIX pods cannot start and keeps crashing. The pod logs show:

- `apisix.lua init_etcd`
- request to `http://apisix-etcd:2379/v3/kv/put` **times out**
- container exits with code 1 → `CrashLoopBackOff`

At the same time, if I run a temporary/debug pod in the same namespace and connect to `apisix-etcd:2379`, the connection works and `etcdctl put/get` succeeds.

So the endpoints are reachable, but APISIX startup (`init_etcd`) still fails due to request timeouts.

### Expected Behavior

_No response_

### Error Logs

_No response_

### Steps to Reproduce
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: apisix
namespace: default
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: apisix
app.kubernetes.io/version: 3.2.0
helm.sh/chart: apisix-1.3.1
annotations:
deployment.kubernetes.io/revision: ""
meta.helm.sh/release-name: apisix
meta.helm.sh/release-namespace: default
spec:
replicas: 2
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
selector:
matchLabels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
template:
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
annotations:
checksum/config: ""
redeploy-timestamp: ""
spec:
serviceAccountName: default
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 30

initContainers:
- name: wait-etcd
image: "/busybox:1.28"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- >
until nc -z apisix-etcd.default.svc.cluster.local 2379; do
echo waiting for etcd `date`;
sleep 2;
done;

containers:
- name: apisix
image: "/apisix:3.14.1-debian"
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 30"]

ports:
- name: http
containerPort: 9080
protocol: TCP
- name: tls
containerPort: 9443
protocol: TCP
- name: admin
containerPort: 9180
protocol: TCP

readinessProbe:
tcpSocket:
port: 9080
initialDelaySeconds: 90
periodSeconds: 30
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 9

resources: {}

volumeMounts:
- name: apisix-config
mountPath: /usr/local/apisix/conf/config.yaml
subPath: config.yaml

# Custom plugins / extra mounted files (all from ConfigMaps)
- name: plugin-cm-1
mountPath: /usr/local/apisix/apisix/plugins/.lua
subPath: .lua
- name: plugin-cm-2
mountPath: /usr/local/apisix/apisix/plugins/.lua
subPath: .lua
- name: plugin-cm-3
mountPath: /usr/local/apisix/apisix/plugins/.lua
subPath: .lua
- name: plugin-cm-4
mountPath: /usr/local/apisix/apisix/plugins/.lua
subPath: .lua
- name: plugin-cm-5
mountPath: /usr/local/apisix/apisix/plugins/.lua
subPath: .lua
- name: extra-file-cm
mountPath: /usr/local/openresty/lualib/resty/.lua
subPath: .lua

volumes:
- name: apisix-config
configMap:
name: apisix
defaultMode: 420

- name: plugin-cm-1
configMap:
name: apisix-plugin-new
defaultMode: 420
- name: plugin-cm-2
configMap:
name: apisix-plugin-new
defaultMode: 420
- name: plugin-cm-3
configMap:
name: apisix-plugin-new
defaultMode: 420
- name: plugin-cm-4
configMap:
name: apisix-plugin-new
defaultMode: 420
- name: plugin-cm-5
configMap:
name: apisix-plugin-new
defaultMode: 420
- name: extra-file-cm
configMap:
name: apisix-plugin-new
defaultMode: 420
```
### Environment

- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):

Contributor guide

Open the contributing guide

Research direction

Start with apisix/cli/apisix.lua and the init_etcd startup path, then compare the mounted config.yaml with the Helm Deployment and its wait-etcd init container. Reproduce the timeout using the supplied Kubernetes manifest and collect the missing APISIX, OpenResty, operating-system, and etcd versions. Done means identifying a reproducible cause and documenting or fixing the startup failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, lua
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.