Graylog2 / Graylog2/graylog2-server
Cache public keys and collector records in AgentTokenService to reduce per-heartbeat overhead
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
Every Collector heartbeat triggers `AgentTokenService.validateAgentToken()` (`AgentTokenService.java:72-107`), which:
1. Calls `collectorInstanceService.findByActiveOrNextFingerprint(fingerprint)` (line 93) — a MongoDB query on every heartbeat
2. Calls `PemUtils.parseCertificate(certPem)` via `getPublicKey()` (line 131-138) — BouncyCastle ASN.1 parsing + X509 certificate construction on every heartbeat
3. Verifies the Ed25519 JWT signature via `Jwts.parser().parseSignedClaims()` (line 100)
Nothing is cached between heartbeats. At 2000 collectors with a 30-second heartbeat interval, this is ~67 MongoDB round-trips and ~67 PEM certificate parses per second.
The source code contains a TODO acknowledging this at line 92:
```java
// TODO performance this loads the entire collector instance document, which seems excessive
```
JFR profiling at 1900 collectors confirmed:
- MongoDB socket reads: p50=29ms, p99=82ms per query
- The cumulative load from heartbeat authentication causes JWKS fetch timeouts when new collectors try to enroll, creating a ceiling at ~1900 collectors on a single server instance
Caffeine caches (already a project dependency) could be added for public key and collector DTO lookups:
```java
private final Cache publicKeyCache = Caffeine.newBuilder()
.maximumSize(4000)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
private final Cache collectorCache = Caffeine.newBuilder()
.maximumSize(4000)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
```
In the `keyLocator` callback, check caches before querying MongoDB:
- Cache hit: return cached PublicKey and CollectorInstanceDTO directly
- Cache miss: query MongoDB, parse PEM, populate caches, then return
The Ed25519 signature verification still runs every request (each JWT is unique), but with a cached public key instead of re-parsing the certificate.
Tested with a patched build at 2000 collectors:
- MongoDB queries for authentication dropped from ~67/sec to ~1/sec (cache misses only)
- Failure threshold moved from 1900 to 2000 collectors
- Memory overhead: ~8-10MB for both caches at 2000 entries (~4-5KB per collector)
- Graylog CPU unchanged (307m at 2000 collectors)
## Steps to Reproduce (for bugs)
1. Deploy Graylog 7.1.0-beta.1 with a single server instance
2. Deploy 1900+ collector pods as a Kubernetes Deployment with 30s heartbeat
3. Ramp collectors in 100-pod increments
4. Observe JWKS fetch timeouts and collectors going offline at ~1900
## Context
Running 2000 Graylog collectors on a 4-node K3s cluster. JFR profiling and source code analysis identified the per-heartbeat MongoDB query + PEM parse as the primary bottleneck preventing scaling beyond 1900 collectors. The cache eliminates ~98% of MongoDB authentication queries and moves the ceiling to 2000+.
## Why?
Collector heartbeat authentication does not have to perform a MongoDB round-trip and full PEM certificate parse on every request.
Public keys and collector records could be cached in memory with a reasonable TTL, since they rarely change (certificates have a 1-year lifetime).
## Your Environment
* Graylog Version: 7.1.0-beta.1 (Enterprise)
* Java Version: OpenJDK 21.0.10 (Temurin)
* OpenSearch Version: 2.x (single node)
* MongoDB Version: 8.x (single node)
* Operating System: Debian 13 (Trixie), K3s v1.35.3
Contributor guide
Assessment
This issue has not been assessed yet.