apache / apache/doris-operator
Context deadline exceeded
- Dominant language
- Go
- Stars
- 138
- Forks
- 72
- Avg merge
- 5d 12h
- Merged PRs (30d)
- 3
Description
Hi,I deployed only one node doris-operator(version 1.6.0) in k8s,and sometimes doris-operator would restart,the error as below:
E0914 01:56:25.370009 1 leaderelection.go:330] error retrieving resource lock grass-roots-wannian/e1370669.selectdb.com: Get "https://10.68.0.1:443/apis/coordination.k8s.io/v1/namespaces/grass-roots-wannian/leases/e1370669.selectdb.com": context deadline exceeded
I0914 01:56:25.370168 1 leaderelection.go:283] failed to renew lease grass-roots-wannian/e1370669.selectdb.com: timed out waiting for the condition
2024-09-14T01:56:25Z INFO Stopping and waiting for non leader election runnables
2024-09-14T01:56:25Z INFO Stopping and waiting for leader election runnables
2024-09-14T01:56:25Z INFO Stopping and waiting for caches
2024-09-14T01:56:25Z INFO Stopping and waiting for webhooks
2024-09-14T01:56:25Z INFO Wait completed, proceeding to shutdown the manager
2024-09-14T01:56:25Z INFO Shutdown signal received, waiting for all workers to finish {"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster"}
2024-09-14T01:56:25Z INFO All workers finished {"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster"}
2024-09-14T01:56:25Z DEBUG events doris-operator-5bfbffc6fd-jkv87_80c6fd10-a3a6-460b-86c6-79a92d37009b stopped leading {"type": "Normal", "object": {"kind":"Lease","namespace":"grass-roots-wannian","name":"e1370669.selectdb.com","uid":"9370b444-e38a-44ff-8c77-6420d2e72243","apiVersion":"coordination.k8s.io/v1","resourceVersion":"12311346"}, "reason": "LeaderElection"}
2024-09-14T01:56:25Z ERROR setup problem running manager {"error": "leader election lost"}
main.main
/workspace/cmd/operator/main.go:155
runtime.main
/usr/local/go/src/runtime/proc.go:250
Would you please help me to check it,I have no idea,thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at cmd/operator/main.go:155 and review the shutdown path shown after leader election is lost. Reproduce or investigate the reported context deadline exceeded and determine why the single operator restarts; done means documenting the cause and providing a verified fix or clear next steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100