Graylog2 / Graylog2/graylog2-server

Problem with async cache invalidation in MongoIndexSetRegistry

Open
#4,625 1 comment 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

The [IndexSetRegistry#get(String)](https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/indexer/IndexSetRegistry.java#L40) method should return the new index set when called in an event bus handler that handles the index set creation.

## Current Behavior

Depending on the event handler registration order, calling `IndexSetRegistry#get(String)` in an event handler for the `IndexSetCreatedEvent` might return an empty result.

```java
@Subscribe
public void handleIndexSetCreation(final IndexSetCreatedEvent event) {
final String indexSetId = event.indexSet().id();
final Optional optionalIndexSet = indexSetRegistry.get(indexSetId);

if (optionalIndexSet.isPresent()) {
// ...
} else {
// This sometimes happens, depending on the order of the event handler
}
}
```

This should not happen. When handling an `IndexSetCreatedEvent` the newly created index set should be accessible via the index set registry.

## Possible Solution

Fix cache in `MongoIndexSetRegistry`.

## Steps to Reproduce (for bugs)

It might not be possible to successfully reproduce this because the issue depends on the event handler registration order in `EventBus`!

1. Register event handler for `IndexSetCreatedEvent`
2. Create new index set
3. `IndexSetRegistry#get(String)` might return an empty result for the newly created index set

## Context

The [MongoIndexSetRegistry](https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/indexer/MongoIndexSetRegistry.java) uses an in-memory cache to reduce load on the database. Each Graylog node runs one instance of that cache and thus relies on cluster events to invalidate the cache.

The problem is that the cache is using the same event infrastructure as other code. That means a custom event handler for the `IndexSetCreatedEvent` might run before the event handler in the cache that invalidates the cache.

When the custom event handler calls `IndexSetRegistry#get(String)` with the ID of the new index set before the cache event handler has run, it gets an empty result because the new index set is not in the cache yet.

https://github.com/Graylog2/graylog2-server/blob/77d03b0c8d038ccc98e95d1a4afe585b89fc6570/graylog2-server/src/main/java/org/graylog2/indexer/MongoIndexSetRegistry.java#L52-L82

## Your Environment

* Graylog Version: 3.0.0-SNAPSHOT (3ebaf88e9)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.