Graylog2 / Graylog2/graylog2-server

Race in GeoIpDbFileChangeMonitorService on startup — concurrent downloads fail with FileAlreadyExistsException, node starts without geo enrichment

Open
#26,880 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

When `pull_from_cloud` (S3) is configured for the Geo-Location Processor, a starting node downloads the database files once and geo enrichment is functional from the moment the node begins processing messages.

## Current Behavior

On startup with an empty download directory, the initial download reliably fails with:

```
ERROR [S3GeoIpFileService] Failed to retrieve files.
java.nio.file.FileAlreadyExistsException: /data/geolocation/temp-standard_location-from-cloud.mmdb
ERROR [GeoIpDbFileChangeMonitorService] Failed to pull new Geo-Location Processor database files from the cloud. Geo-Location Processor will not be functional on this node.
```

The node finishes booting and **processes messages without geo enrichment** until the next refresh tick (~1 minute with a short `refresh_interval`) succeeds. A "Geo-Location Processor cloud Sync Failure" notification is raised.

In our tests this was deterministic on cold starts: 5/5 nodes across two separate cold starts of a 3-node cluster hit it.

The follow-on problem: **Success is silent.** Successful downloads are logged at DEBUG only, so from INFO-level logs a recovered node is indistinguishable from a broken one.

## Cause (from reading the code)

`GeoIpDbFileChangeMonitorService.updateConfiguration()` is triggered both from the service's `startUp()` and from the `ClusterConfigChangedEvent` handler, with no synchronization between them. Both invocations run `S3GeoIpFileService` downloads concurrently against the **hardcoded temp file names** (`temp-standard_location-from-cloud.mmdb` / `temp-asn-from-cloud.mmdb`); the AWS SDK's `S3TransferManager` refuses to write to an existing file, so the slower invocation dies with `FileAlreadyExistsException`. On a cold start (empty dir) the two invocations overlap for the whole download duration, which is why it reproduces ~100% there.

Possible fixes: serialize `updateConfiguration()` (lock/`synchronized`), or download to a unique temp name (suffix with UUID/thread id) and atomically move into place.

## Steps to Reproduce

1. Configure Geo-Location Processor with `pull_from_cloud` = S3, IPINFO vendor, valid `s3://` city/ASN paths.
2. Start a node with an empty `data_dir/geolocation` (fresh volume) — a multi-node cluster makes it more visible but a single node reproduces it too.
3. Observe the ERROR above during boot; the files appear one refresh tick later.

## Your Environment

* Graylog Version: 7.1.7
* OpenSearch Version: 2.x
* MongoDB Version: 8.0

Contributor guide

Open the contributing guide

Research direction

Start by reading GeoIpDbFileChangeMonitorService.updateConfiguration(), including its startUp() and ClusterConfigChangedEvent call paths, then inspect S3GeoIpFileService and the concurrent download behavior. Reproduce with pull_from_cloud and an empty geolocation directory; done means startup downloads reliably without FileAlreadyExistsException, geo enrichment is available immediately, and successful recovery is visible at INFO level.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.