Graylog2 / Graylog2/collector

Stagger collector startup to prevent enrollment thundering herd

Open Beginner friendly
#8 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1
Forks
0
Avg merge
3d 1h
Merged PRs (30d)
7

Description

## Expected Behavior

When many Collector pods start simultaneously (e.g., Kubernetes Deployment scale-up) enrollment requests should be spread over a time window to avoid overwhelming the Graylog server. Each collector should wait a random delay (e.g., 0-30 seconds) before initiating its OpAMP connection and enrollment.

## Current Behavior

When a Kubernetes Deployment scales from 0 to N collectors, all N pods start within seconds. Each pod immediately calls `sup.Start()` (`cmd/supervisor/main.go:168`), which initiates the OpAMP connection and enrollment. With 100+ pods starting simultaneously, the Graylog server receives 100+ concurrent JWKS fetch and enrollment requests.

This can cause:
- JWKS endpoint timeouts (`context deadline exceeded` on `/.well-known/jwks.json`).
- Server-side CPU spikes from concurrent Ed25519 certificate signing.
- MongoDB write contention on `collector_instances` collection.

Pods that fail enrollment crash:
```go
Error: failed to start supervisor: authentication initialization failed:
enrollment preparation failed: failed to fetch JWKS: Get
"https://graylog.example.com/.well-known/jwks.json":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```

Crashed pods restart and retry at the same time, which cause a thundering herd of load on a Graylog server.

JFR profiling of a Graylog server during a 100-pod enrollment burst showed:
- HK2 `ServiceLocatorImpl.addConfiguration()`: 26% of CPU (each enrollment triggers a full DI container cache invalidation).
- Ed25519 certificate signing: 23% of CPU.

## Possible Solution

Add a random startup delay before `sup.Start()` in `cmd/supervisor/main.go`:

```go
jitter := time.Duration(rand.IntN(30)) * time.Second
logger.Info("Staggering startup to spread enrollment load", zap.Duration("delay", jitter))
select {
case <-time.After(jitter):
case <-ctx.Done():
logger.Info("Shutdown before startup delay completed")
return
}
```

This spreads enrollment requests over a 30-second window. For a 100-pod scale-up, this means ~3 enrollments per second instead of 100 simultaneous.

The `select` with `ctx.Done()` ensures the delay is cancelled immediately if the pod receives a shutdown signal during the wait.

Tested with a patched build:
- Without stagger: 8-14 restarts during 0→1850 ramp
- With stagger: **0 restarts** during 0→2000 ramp (60s heartbeat)

## Steps to Reproduce (for bugs)
1. Deploy a Graylog server node/pod.
2. Deploy 100+ collector pods simultaneously: `kubectl scale deployment graylog-collector --replicas=100`
- Note: Depending on hardware capacity of the host, more pods might be required to reproduce the problem.
3. Observe: many pods crash with JWKS timeout, restart, create another burst.
4. With staggered startup: pods spread enrollment over 30 seconds, all succeed on first attempt.
## Context

Running 2000 Graylog collectors on a 4-node K3s cluster. Without staggered startup, scaling up by 100-500 pods (or more) can cause enrollment storms that overwhelm the Graylog server. The staggered startup eliminates all enrollment failures during scale-up, making the system reliable at high collector counts.

## Your Environment

* Graylog Version: 7.1.0-beta.1 (Enterprise)
* Collector Version: 2.0.0-SNAPSHOT (v2 branch)
* Operating System: Debian 13 (Trixie), K3s v1.35.3
* Deployment: Kubernetes Deployment (not StatefulSet)

## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at cmd/supervisor/main.go:168, where sup.Start() initiates the OpAMP connection and enrollment, and review how the startup context is handled. Add the cancellable randomized delay before startup, then verify that scale-up enrollment requests are spread over the window and shutdown interrupts the wait.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.