Graylog2 / Graylog2/graylog2-server
Race in GeoIpDbFileChangeMonitorService on startup — concurrent downloads fail with FileAlreadyExistsException, node starts without geo enrichment
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
When `pull_from_cloud` (S3) is configured for the Geo-Location Processor, a starting node downloads the database files once and geo enrichment is functional from the moment the node begins processing messages.
## Current Behavior
On startup with an empty download directory, the initial download reliably fails with:
```
ERROR [S3GeoIpFileService] Failed to retrieve files.
java.nio.file.FileAlreadyExistsException: /data/geolocation/temp-standard_location-from-cloud.mmdb
ERROR [GeoIpDbFileChangeMonitorService] Failed to pull new Geo-Location Processor database files from the cloud. Geo-Location Processor will not be functional on this node.
```
The node finishes booting and **processes messages without geo enrichment** until the next refresh tick (~1 minute with a short `refresh_interval`) succeeds. A "Geo-Location Processor cloud Sync Failure" notification is raised.
In our tests this was deterministic on cold starts: 5/5 nodes across two separate cold starts of a 3-node cluster hit it.
The follow-on problem: **Success is silent.** Successful downloads are logged at DEBUG only, so from INFO-level logs a recovered node is indistinguishable from a broken one.
## Cause (from reading the code)
`GeoIpDbFileChangeMonitorService.updateConfiguration()` is triggered both from the service's `startUp()` and from the `ClusterConfigChangedEvent` handler, with no synchronization between them. Both invocations run `S3GeoIpFileService` downloads concurrently against the **hardcoded temp file names** (`temp-standard_location-from-cloud.mmdb` / `temp-asn-from-cloud.mmdb`); the AWS SDK's `S3TransferManager` refuses to write to an existing file, so the slower invocation dies with `FileAlreadyExistsException`. On a cold start (empty dir) the two invocations overlap for the whole download duration, which is why it reproduces ~100% there.
Possible fixes: serialize `updateConfiguration()` (lock/`synchronized`), or download to a unique temp name (suffix with UUID/thread id) and atomically move into place.
## Steps to Reproduce
1. Configure Geo-Location Processor with `pull_from_cloud` = S3, IPINFO vendor, valid `s3://` city/ASN paths.
2. Start a node with an empty `data_dir/geolocation` (fresh volume) — a multi-node cluster makes it more visible but a single node reproduces it too.
3. Observe the ERROR above during boot; the files appear one refresh tick later.
## Your Environment
* Graylog Version: 7.1.7
* OpenSearch Version: 2.x
* MongoDB Version: 8.0
Contributor guide
Research direction
Start by reading GeoIpDbFileChangeMonitorService.updateConfiguration(), including its startUp() and ClusterConfigChangedEvent call paths, then inspect S3GeoIpFileService and the concurrent download behavior. Reproduce with pull_from_cloud and an empty geolocation directory; done means startup downloads reliably without FileAlreadyExistsException, geo enrichment is available immediately, and successful recovery is visible at INFO level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100