apache / apache/doris-operator

Context deadline exceeded

Open
#246 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
138
Forks
72
Avg merge
5d 12h
Merged PRs (30d)
3

Description

Hi,I deployed only one node doris-operator(version 1.6.0) in k8s,and sometimes doris-operator would restart,the error as below:

E0914 01:56:25.370009 1 leaderelection.go:330] error retrieving resource lock grass-roots-wannian/e1370669.selectdb.com: Get "https://10.68.0.1:443/apis/coordination.k8s.io/v1/namespaces/grass-roots-wannian/leases/e1370669.selectdb.com": context deadline exceeded
I0914 01:56:25.370168 1 leaderelection.go:283] failed to renew lease grass-roots-wannian/e1370669.selectdb.com: timed out waiting for the condition
2024-09-14T01:56:25Z INFO Stopping and waiting for non leader election runnables
2024-09-14T01:56:25Z INFO Stopping and waiting for leader election runnables
2024-09-14T01:56:25Z INFO Stopping and waiting for caches
2024-09-14T01:56:25Z INFO Stopping and waiting for webhooks
2024-09-14T01:56:25Z INFO Wait completed, proceeding to shutdown the manager
2024-09-14T01:56:25Z INFO Shutdown signal received, waiting for all workers to finish {"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster"}
2024-09-14T01:56:25Z INFO All workers finished {"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster"}
2024-09-14T01:56:25Z DEBUG events doris-operator-5bfbffc6fd-jkv87_80c6fd10-a3a6-460b-86c6-79a92d37009b stopped leading {"type": "Normal", "object": {"kind":"Lease","namespace":"grass-roots-wannian","name":"e1370669.selectdb.com","uid":"9370b444-e38a-44ff-8c77-6420d2e72243","apiVersion":"coordination.k8s.io/v1","resourceVersion":"12311346"}, "reason": "LeaderElection"}
2024-09-14T01:56:25Z ERROR setup problem running manager {"error": "leader election lost"}
main.main
/workspace/cmd/operator/main.go:155
runtime.main
/usr/local/go/src/runtime/proc.go:250

Would you please help me to check it,I have no idea,thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at cmd/operator/main.go:155 and review the shutdown path shown after leader election is lost. Reproduce or investigate the reported context deadline exceeded and determine why the single operator restarts; done means documenting the cause and providing a verified fix or clear next steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.