Graylog2 / Graylog2/graylog2-server

Cache public keys and collector records in AgentTokenService to reduce per-heartbeat overhead

Open
#25,716 0 comments 0 reactions 0 assignees View on GitHub
collector
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

Every Collector heartbeat triggers `AgentTokenService.validateAgentToken()` (`AgentTokenService.java:72-107`), which:

1. Calls `collectorInstanceService.findByActiveOrNextFingerprint(fingerprint)` (line 93) — a MongoDB query on every heartbeat
2. Calls `PemUtils.parseCertificate(certPem)` via `getPublicKey()` (line 131-138) — BouncyCastle ASN.1 parsing + X509 certificate construction on every heartbeat
3. Verifies the Ed25519 JWT signature via `Jwts.parser().parseSignedClaims()` (line 100)

Nothing is cached between heartbeats. At 2000 collectors with a 30-second heartbeat interval, this is ~67 MongoDB round-trips and ~67 PEM certificate parses per second.

The source code contains a TODO acknowledging this at line 92:
```java
// TODO performance this loads the entire collector instance document, which seems excessive
```

JFR profiling at 1900 collectors confirmed:
- MongoDB socket reads: p50=29ms, p99=82ms per query
- The cumulative load from heartbeat authentication causes JWKS fetch timeouts when new collectors try to enroll, creating a ceiling at ~1900 collectors on a single server instance

Caffeine caches (already a project dependency) could be added for public key and collector DTO lookups:

```java
private final Cache publicKeyCache = Caffeine.newBuilder()
.maximumSize(4000)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();

private final Cache collectorCache = Caffeine.newBuilder()
.maximumSize(4000)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
```

In the `keyLocator` callback, check caches before querying MongoDB:
- Cache hit: return cached PublicKey and CollectorInstanceDTO directly
- Cache miss: query MongoDB, parse PEM, populate caches, then return

The Ed25519 signature verification still runs every request (each JWT is unique), but with a cached public key instead of re-parsing the certificate.

Tested with a patched build at 2000 collectors:
- MongoDB queries for authentication dropped from ~67/sec to ~1/sec (cache misses only)
- Failure threshold moved from 1900 to 2000 collectors
- Memory overhead: ~8-10MB for both caches at 2000 entries (~4-5KB per collector)
- Graylog CPU unchanged (307m at 2000 collectors)

## Steps to Reproduce (for bugs)
1. Deploy Graylog 7.1.0-beta.1 with a single server instance
2. Deploy 1900+ collector pods as a Kubernetes Deployment with 30s heartbeat
3. Ramp collectors in 100-pod increments
4. Observe JWKS fetch timeouts and collectors going offline at ~1900

## Context

Running 2000 Graylog collectors on a 4-node K3s cluster. JFR profiling and source code analysis identified the per-heartbeat MongoDB query + PEM parse as the primary bottleneck preventing scaling beyond 1900 collectors. The cache eliminates ~98% of MongoDB authentication queries and moves the ceiling to 2000+.

## Why?

Collector heartbeat authentication does not have to perform a MongoDB round-trip and full PEM certificate parse on every request.

Public keys and collector records could be cached in memory with a reasonable TTL, since they rarely change (certificates have a 1-year lifetime).

## Your Environment

* Graylog Version: 7.1.0-beta.1 (Enterprise)
* Java Version: OpenJDK 21.0.10 (Temurin)
* OpenSearch Version: 2.x (single node)
* MongoDB Version: 8.x (single node)
* Operating System: Debian 13 (Trixie), K3s v1.35.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.