Graylog2 / Graylog2/graylog2-server
StaticFieldFilter and ExtractorFilter create duplicate instances due to missing singleton scope
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Problem
`StaticFieldFilter` and `ExtractorFilter` are instantiated once per process buffer thread instead of once per server. Each instance independently registers itself on the Guava `EventBus`, causing every input lifecycle event (`InputCreated`, `InputUpdated`, `InputDeleted`) to be handled N times instead of once — where N equals the `processbuffer_processors` count.
### Root cause
The instance multiplication chain:
1. `ProcessBuffer` creates N `ProcessBufferProcessor` instances (N = `processbuffer_processors`, default: `round(cores * 0.36 + 0.625)`)
2. Each injects `OrderedMessageProcessors`, bound `in(Scopes.NO_SCOPE)` (`MessageProcessorModule:30`)
3. Each `OrderedMessageProcessors` injects `Set`, creating a new `MessageFilterChainProcessor`
4. Each `MessageFilterChainProcessor` injects `Set`, creating new `StaticFieldFilter` and `ExtractorFilter`
instances
5. Each filter registers itself on `serverEventBus` in its constructor
Neither `StaticFieldFilter` nor `ExtractorFilter` is annotated `@Singleton`, and the multibinder binding in
`MessageFilterBindings` does not specify a scope.
There is an existing TODO in `StaticFieldFilter:62`:
```java
// TODO: This class needs lifecycle management to avoid leaking objects in the EventBus
serverEventBus.register(this);
```
### Observable symptoms
On a 12-core machine (`processbuffer_processors` = 5):
- Deleting a single input produces 5 identical log lines:
```
DEBUG: o.g.filters.StaticFieldFilter - Removing input from static fields cache
```
- Creating or updating an input triggers 5 identical MongoDB queries for static fields (and 5 for extractors)
## Suggested fix
Extract the shared cache and event bus handling into a singleton service. The per-thread filter instances should reference the shared cache instead of maintaining their own.
For `StaticFieldFilter`:
1. Create a `@Singleton` class (e.g., `StaticFieldsCache`) that:
- Owns the `ConcurrentMap>>` cache
- Registers on `serverEventBus` and handles `InputCreated`, `InputUpdated`, `InputDeleted` events
- Loads static fields from `InputService` on demand and at startup
2. Modify `StaticFieldFilter` to inject the singleton cache and use it in `filter()`, removing its own event bus
registration and cache
Apply the same pattern to `ExtractorFilter`.
### Files to modify
- `graylog2-server/src/main/java/org/graylog2/filters/StaticFieldFilter.java`
- `graylog2-server/src/main/java/org/graylog2/filters/ExtractorFilter.java`
- `graylog2-server/src/main/java/org/graylog2/bindings/MessageFilterBindings.java` (add new singleton bindings)
## Expected benefit
| Aspect | Before (N = processbuffer_processors) | After |
|--------|---------------------------------------|-------|
| Event bus subscribers per filter type | N | 1 |
| MongoDB queries per input lifecycle event | N | 1 |
| MongoDB queries at startup (M inputs) | N × M | M |
| Duplicate log lines per event | N | 1 |
| Static field cache memory | N copies | 1 copy |
The primary benefit is eliminating redundant MongoDB queries during input lifecycle events and at server startup. For a deployment with 100 inputs and `processbuffer_processors=5`, startup static field loading drops from 500 queries to 100 (same for extractor loading).
Functional correctness is not affected — the current behavior produces correct results, it just does so with unnecessary duplication.
Contributor guide
Assessment
This issue has not been assessed yet.