apache / apache/apisix-helm-chart

request help: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex

Open
#290 8 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Go Template
Stars
289
Forks
282
Avg merge
2d 19h
Merged PRs (30d)
4

Description

k8s版本: aws 宁夏区 eks
kubectl version --short
Client Version: v1.19.6-eks-49a6c0
Server Version: v1.20.15-eks-0d102a7

apisix 版本:
```
$ helm list -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apisix ingress-apisix 1 2022-05-18 16:17:51.725532 +0800 CST deployed apisix-0.9.3 2.13.1
```
apache/apisix:2.13.1-alpine
apache/apisix-ingress-controller:1.4.1
docker.io/bitnami/etcd:3.4.18-debian-10-r14

pod:
```
$ kubectl get po -o wide -n ingress-apisix
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apisix-78f5cb9b84-82czl 1/1 Running 0 16h 172.19.115.235 ip-172-19-113-187.cn-northwest-1.compute.internal
apisix-78f5cb9b84-qczbv 1/1 Running 0 16h 172.19.117.215 ip-172-19-112-190.cn-northwest-1.compute.internal
apisix-dashboard-5465fb4df7-9z8xg 1/1 Running 4 16h 172.19.116.190 ip-172-19-124-247.cn-northwest-1.compute.internal
apisix-etcd-0 1/1 Running 0 16h 172.19.125.197 ip-172-19-119-174.cn-northwest-1.compute.internal
apisix-etcd-1 1/1 Running 0 16h 172.19.98.160 ip-172-19-124-249.cn-northwest-1.compute.internal
apisix-etcd-2 1/1 Running 0 16h 172.19.119.222 ip-172-19-117-72.cn-northwest-1.compute.internal
apisix-ingress-controller-87d78d98d-dftls 1/1 Running 0 16h 172.19.97.227 ip-172-19-124-247.cn-northwest-1.compute.internal
```
我们把 apisix-etcd-2所在的k8s节点(ip-172-19-117-72.cn-northwest-1.compute.internal)设置为维护模式,并驱逐该节点的apisix-etcd-2 到其他节点上:
```
$ kubectl drain ip-172-19-117-72.cn-northwest-1.compute.internal --ignore-daemonsets --delete-local-data
node/ip-172-19-117-72.cn-northwest-1.compute.internal cordoned
WARNING: ignoring DaemonSet-managed Pods: default/prometheus-node-exporter-55rbb, kube-system/aws-node-m5xjb, kube-system/efs-csi-node-dlln4, kube-system/kube-proxy-xhjz7
evicting pod ingress-apisix/apisix-etcd-2
pod/apisix-etcd-2 evicted
node/ip-172-19-117-72.cn-northwest-1.compute.internal evicted

```
检查apisix-etcd-2迁移到新k8s节点后的运行状况,apisix-etcd-2在新的k8s节点(ip-172-19-97-147.cn-northwest-1.compute.internal) 一直处在CrashLoopBackOff状态:
```
$ kubectl get po -o wide -n ingress-apisix
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apisix-78f5cb9b84-82czl 1/1 Running 0 17h 172.19.115.235 ip-172-19-113-187.cn-northwest-1.compute.internal
apisix-78f5cb9b84-qczbv 1/1 Running 0 17h 172.19.117.215 ip-172-19-112-190.cn-northwest-1.compute.internal
apisix-dashboard-5465fb4df7-9z8xg 1/1 Running 4 17h 172.19.116.190 ip-172-19-124-247.cn-northwest-1.compute.internal
apisix-etcd-0 1/1 Running 0 17h 172.19.125.197 ip-172-19-119-174.cn-northwest-1.compute.internal
apisix-etcd-1 1/1 Running 0 17h 172.19.98.160 ip-172-19-124-249.cn-northwest-1.compute.internal
apisix-etcd-2 0/1 CrashLoopBackOff 8 16m 172.19.100.35 ip-172-19-97-147.cn-northwest-1.compute.internal
apisix-ingress-controller-87d78d98d-dftls 1/1 Running 0 17h 172.19.97.227 ip-172-19-124-247.cn-northwest-1.compute.internal
```
apisix-etcd-2 的日志:
```
$ kubectl logs apisix-etcd-2 -n ingress-apisix -f
etcd 01:28:08.43
etcd 01:28:08.43 Welcome to the Bitnami etcd container
etcd 01:28:08.43 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 01:28:08.43 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 01:28:08.44
etcd 01:28:08.44 INFO ==> ** Starting etcd setup **
etcd 01:28:08.46 INFO ==> Validating settings in ETCD_* env vars..
etcd 01:28:08.46 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 01:28:08.47 INFO ==> Initializing etcd
etcd 01:28:08.47 INFO ==> Generating etcd config file using env variables
etcd 01:28:08.48 INFO ==> Detected data from previous deployments
etcd 01:28:08.61 INFO ==> Updating member in existing cluster
Error: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the apisix-etcd-2 pod logs and the Helm deployment details for apache/apisix-helm-chart 0.9.3, then inspect how the Bitnami etcd container updates a member after kubectl drain moves it. Compare the failing member state with the two running etcd pods; done means the migrated pod starts normally and the etcd cluster remains healthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm, kubernetes
Domain
databases, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.