apache / apache/apisix-helm-chart
request help: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
- Dominant language
- Go Template
- Stars
- 289
- Forks
- 282
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 4
Description
k8s版本: aws 宁夏区 eks
kubectl version --short
Client Version: v1.19.6-eks-49a6c0
Server Version: v1.20.15-eks-0d102a7
apisix 版本:
```
$ helm list -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apisix ingress-apisix 1 2022-05-18 16:17:51.725532 +0800 CST deployed apisix-0.9.3 2.13.1
```
apache/apisix:2.13.1-alpine
apache/apisix-ingress-controller:1.4.1
docker.io/bitnami/etcd:3.4.18-debian-10-r14
pod:
```
$ kubectl get po -o wide -n ingress-apisix
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apisix-78f5cb9b84-82czl 1/1 Running 0 16h 172.19.115.235 ip-172-19-113-187.cn-northwest-1.compute.internal
apisix-78f5cb9b84-qczbv 1/1 Running 0 16h 172.19.117.215 ip-172-19-112-190.cn-northwest-1.compute.internal
apisix-dashboard-5465fb4df7-9z8xg 1/1 Running 4 16h 172.19.116.190 ip-172-19-124-247.cn-northwest-1.compute.internal
apisix-etcd-0 1/1 Running 0 16h 172.19.125.197 ip-172-19-119-174.cn-northwest-1.compute.internal
apisix-etcd-1 1/1 Running 0 16h 172.19.98.160 ip-172-19-124-249.cn-northwest-1.compute.internal
apisix-etcd-2 1/1 Running 0 16h 172.19.119.222 ip-172-19-117-72.cn-northwest-1.compute.internal
apisix-ingress-controller-87d78d98d-dftls 1/1 Running 0 16h 172.19.97.227 ip-172-19-124-247.cn-northwest-1.compute.internal
```
我们把 apisix-etcd-2所在的k8s节点(ip-172-19-117-72.cn-northwest-1.compute.internal)设置为维护模式,并驱逐该节点的apisix-etcd-2 到其他节点上:
```
$ kubectl drain ip-172-19-117-72.cn-northwest-1.compute.internal --ignore-daemonsets --delete-local-data
node/ip-172-19-117-72.cn-northwest-1.compute.internal cordoned
WARNING: ignoring DaemonSet-managed Pods: default/prometheus-node-exporter-55rbb, kube-system/aws-node-m5xjb, kube-system/efs-csi-node-dlln4, kube-system/kube-proxy-xhjz7
evicting pod ingress-apisix/apisix-etcd-2
pod/apisix-etcd-2 evicted
node/ip-172-19-117-72.cn-northwest-1.compute.internal evicted
```
检查apisix-etcd-2迁移到新k8s节点后的运行状况,apisix-etcd-2在新的k8s节点(ip-172-19-97-147.cn-northwest-1.compute.internal) 一直处在CrashLoopBackOff状态:
```
$ kubectl get po -o wide -n ingress-apisix
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apisix-78f5cb9b84-82czl 1/1 Running 0 17h 172.19.115.235 ip-172-19-113-187.cn-northwest-1.compute.internal
apisix-78f5cb9b84-qczbv 1/1 Running 0 17h 172.19.117.215 ip-172-19-112-190.cn-northwest-1.compute.internal
apisix-dashboard-5465fb4df7-9z8xg 1/1 Running 4 17h 172.19.116.190 ip-172-19-124-247.cn-northwest-1.compute.internal
apisix-etcd-0 1/1 Running 0 17h 172.19.125.197 ip-172-19-119-174.cn-northwest-1.compute.internal
apisix-etcd-1 1/1 Running 0 17h 172.19.98.160 ip-172-19-124-249.cn-northwest-1.compute.internal
apisix-etcd-2 0/1 CrashLoopBackOff 8 16m 172.19.100.35 ip-172-19-97-147.cn-northwest-1.compute.internal
apisix-ingress-controller-87d78d98d-dftls 1/1 Running 0 17h 172.19.97.227 ip-172-19-124-247.cn-northwest-1.compute.internal
```
apisix-etcd-2 的日志:
```
$ kubectl logs apisix-etcd-2 -n ingress-apisix -f
etcd 01:28:08.43
etcd 01:28:08.43 Welcome to the Bitnami etcd container
etcd 01:28:08.43 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 01:28:08.43 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 01:28:08.44
etcd 01:28:08.44 INFO ==> ** Starting etcd setup **
etcd 01:28:08.46 INFO ==> Validating settings in ETCD_* env vars..
etcd 01:28:08.46 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 01:28:08.47 INFO ==> Initializing etcd
etcd 01:28:08.47 INFO ==> Generating etcd config file using env variables
etcd 01:28:08.48 INFO ==> Detected data from previous deployments
etcd 01:28:08.61 INFO ==> Updating member in existing cluster
Error: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the apisix-etcd-2 pod logs and the Helm deployment details for apache/apisix-helm-chart 0.9.3, then inspect how the Bitnami etcd container updates a member after kubectl drain moves it. Compare the failing member state with the two running etcd pods; done means the migrated pod starts normally and the etcd cluster remains healthy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, kubernetes
- Domain
- databases, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100