temporalio / temporalio/temporal
Check DB Connectivity in Liveness probe
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
In K8s environment there are situations when DB passwords and certificates are renewed.
When that happens the current worker, frontend, matching and history behavior is to fail silently and log the error.
The passwords are stored in K8s secrets and loaded as environment variables, in other words, a pod restart would resolve the issue.
Describe the solution you'd like
Check DB connectivity in liveness probe.
If the DB password is changed the temporal pods would restart and load the new secret.
Describe alternatives you've considered
Using tctl namespace list as liveness probe exec command, but it can't connect to the localhost even if the correct port is specified.
Plus, replacing the existing liveness probe feels hacky.
Additional context
An example of the tctl n l failure when run inside the history pod.
temporal-history-67f566466-qz8n8:/etc/temporal$ tctl --address localhost:7234 n l
Error: Error when list namespaces info
Error Details: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the liveness probe definitions for the worker, frontend, matching, and history pods. Reproduce the TLS or database-authentication failure described in the issue and determine how each probe currently behaves. Done means a database connectivity failure causes the affected Temporal pod to restart and reload its Kubernetes secrets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100