Stagger collector startup to prevent enrollment thundering herd
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1
- Forks
- 0
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 7
Description
## Expected Behavior
When many Collector pods start simultaneously (e.g., Kubernetes Deployment scale-up) enrollment requests should be spread over a time window to avoid overwhelming the Graylog server. Each collector should wait a random delay (e.g., 0-30 seconds) before initiating its OpAMP connection and enrollment.
## Current Behavior
When a Kubernetes Deployment scales from 0 to N collectors, all N pods start within seconds. Each pod immediately calls `sup.Start()` (`cmd/supervisor/main.go:168`), which initiates the OpAMP connection and enrollment. With 100+ pods starting simultaneously, the Graylog server receives 100+ concurrent JWKS fetch and enrollment requests.
This can cause:
- JWKS endpoint timeouts (`context deadline exceeded` on `/.well-known/jwks.json`).
- Server-side CPU spikes from concurrent Ed25519 certificate signing.
- MongoDB write contention on `collector_instances` collection.
Pods that fail enrollment crash:
```go
Error: failed to start supervisor: authentication initialization failed:
enrollment preparation failed: failed to fetch JWKS: Get
"https://graylog.example.com/.well-known/jwks.json":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```
Crashed pods restart and retry at the same time, which cause a thundering herd of load on a Graylog server.
JFR profiling of a Graylog server during a 100-pod enrollment burst showed:
- HK2 `ServiceLocatorImpl.addConfiguration()`: 26% of CPU (each enrollment triggers a full DI container cache invalidation).
- Ed25519 certificate signing: 23% of CPU.
## Possible Solution
Add a random startup delay before `sup.Start()` in `cmd/supervisor/main.go`:
```go
jitter := time.Duration(rand.IntN(30)) * time.Second
logger.Info("Staggering startup to spread enrollment load", zap.Duration("delay", jitter))
select {
case <-time.After(jitter):
case <-ctx.Done():
logger.Info("Shutdown before startup delay completed")
return
}
```
This spreads enrollment requests over a 30-second window. For a 100-pod scale-up, this means ~3 enrollments per second instead of 100 simultaneous.
The `select` with `ctx.Done()` ensures the delay is cancelled immediately if the pod receives a shutdown signal during the wait.
Tested with a patched build:
- Without stagger: 8-14 restarts during 0→1850 ramp
- With stagger: **0 restarts** during 0→2000 ramp (60s heartbeat)
## Steps to Reproduce (for bugs)
1. Deploy a Graylog server node/pod.
2. Deploy 100+ collector pods simultaneously: `kubectl scale deployment graylog-collector --replicas=100`
- Note: Depending on hardware capacity of the host, more pods might be required to reproduce the problem.
3. Observe: many pods crash with JWKS timeout, restart, create another burst.
4. With staggered startup: pods spread enrollment over 30 seconds, all succeed on first attempt.
## Context
Running 2000 Graylog collectors on a 4-node K3s cluster. Without staggered startup, scaling up by 100-500 pods (or more) can cause enrollment storms that overwhelm the Graylog server. The staggered startup eliminates all enrollment failures during scale-up, making the system reliable at high collector counts.
## Your Environment
* Graylog Version: 7.1.0-beta.1 (Enterprise)
* Collector Version: 2.0.0-SNAPSHOT (v2 branch)
* Operating System: Debian 13 (Trixie), K3s v1.35.3
* Deployment: Kubernetes Deployment (not StatefulSet)
## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at cmd/supervisor/main.go:168, where sup.Start() initiates the OpAMP connection and enrollment, and review how the startup context is handled. Add the cancellable randomized delay before startup, then verify that scale-up enrollment requests are spread over the window and shutdown interrupts the wait.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100