[🐛 Bug]: selenium/node-kubernetes pod restarts automatically

What happened?

Summary

The test-node-kubernetes pod is restarting automatically without any manual intervention. The restart is observed even after increasing the liveness probe timeoutSeconds from 1 second to 10 seconds.

---

Environment

• Namespace: spr-ops
• Deployment: test-node-kubernetes
• Component: Selenium Grid Kubernetes Node

---

Current Status

kubectl get pods -n spr-ops | grep test

test-node-kubernetes-85485bd7fc-82sx6    1/1 Running   4 (118m ago)   15h
test-selenium-hub-84f5f85bfb-7pdtb       1/1 Running   0              9h

The pod has restarted 4 times within the last 15 hours.

---

Findings

1. Container Exit Code

Last State:
  Terminated
  Reason: Error
  Exit Code: 143

Exit Code 143 indicates the container received a SIGTERM signal (128 + 15), which means Kubernetes gracefully terminated the container instead of the application crashing on its own.

---

2. Container Logs

The logs consistently show a graceful shutdown sequence.

WARN received SIGTERM indicating exit request
INFO waiting for selenium-grid-kubernetes to die
WARN stopped: selenium-grid-kubernetes (terminated by SIGTERM)
Shutdown complete

This indicates:

• The application is not crashing unexpectedly.
• Kubernetes is explicitly sending a SIGTERM signal.
• The application performs a graceful shutdown after receiving the termination signal.

---

3. Liveness Probe Configuration

Current configuration:

livenessProbe:
  httpGet:
    path: /readyz
    port: 5555
  timeoutSeconds: 1
  periodSeconds: 10
  failureThreshold: 3

This configuration means Kubernetes:

• Performs a health check every 10 seconds.
• Waits only 1 second for the /readyz endpoint to respond.
• Restarts the container after 3 consecutive failures (approximately 30 seconds).

This behavior is consistent with the observed SIGTERM-based shutdown.

---

Change Attempted

To verify whether the aggressive timeout was causing false liveness failures, the following change was made:

timeoutSeconds: 10

However, the pod continued to restart, indicating that increasing the timeout alone did not resolve the issue.

---

Expected Behavior

The test-node-kubernetes pod should remain healthy and continue running unless:

• the application becomes unhealthy,
• the node is drained,
• a deployment rollout occurs, or
• another Kubernetes lifecycle event intentionally restarts the pod.

---

Actual Behavior

The pod receives a SIGTERM from Kubernetes and restarts automatically, despite increasing the liveness probe timeout.

---

Impact

• Unexpected Selenium node restarts.
• Existing browser sessions may be interrupted.
• Potential instability in Selenium Grid during test execution.

---

Investigation Completed

• Verified restart count using kubectl get pods.
• Verified container exit code (143 / SIGTERM).
• Reviewed container logs confirming graceful shutdown.
• Reviewed liveness probe configuration.
• Increased timeoutSeconds from 1 to 10.
• Restart issue still persists.

---

Next Steps / Assistance Required

Further investigation is required to determine what is triggering the SIGTERM.

Command used to start Selenium Grid with Docker (or Kubernetes)

kubectl deploy test-selenium-hub -n spr-ops

Relevant log output

WARN received SIGTERM indicating exit request
INFO waiting for selenium-grid-kubernetes to die
WARN stopped: selenium-grid-kubernetes (terminated by SIGTERM)
Shutdown complete

Operating System

mac

Docker Selenium version (image tag)

selenium/node-kubernetes:4.46.0-20260707

Selenium Grid chart version (chart version)

No response

Start with the test-node-kubernetes pod in namespace spr-ops using kubectl get pods and inspect its last terminated state, exit code, logs, and livenessProbe configuration. Trace which Kubernetes lifecycle event is sending SIGTERM despite timeoutSeconds being 10. Done means identifying the trigger and keeping the pod running during normal operation without interrupting browser sessions.