Graylog2 / Graylog2/graylog2-server
Data Node Fails to start if available disk space is less than `node_search_cache_size`, despite used space being consumed by the cache itself
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Data Node Fails to start if available disk space is less than `node_search_cache_size`, despite used space being consumed by the cache itself.
The size of the cache should be subtracted from the used space when checking for available space.
The current logic may require space for the 2x the cache size.
## Expected Behavior
Graylog Data Node can start if the available space less the existing cache size is large enough to fit the configured cache size (`node_search_cache_size`).
## Current Behavior
Graylog Data Node cannot start if there the amount of free space is less than `node_search_cache_size` despite the existing cache size consuming the same amount of space as `node_search_cache_size`.
## Possible Solution
Account for the existing cache size when calculating available space.
## Steps to Reproduce (for bugs)
1.
2.
3.
4.
## Context
Ran into this when my Data Node used exclusively for warm tier failed to start:
```
2025-11-24T15:46:10.916Z ERROR [DatanodeBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:765)
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:586)
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:300)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.startCommand(DatanodeBootstrap.java:138)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:382)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
Caused by: org.graylog.datanode.configuration.OpensearchConfigurationException: There is not enough usable space for the node search cache. Your system has only 36gb available.
Either decrease node_search_cache_size configuration or make sure that datanode has enough free disk space.
Data directory: /var/lib/opensearch, current node_search_cache_size: 50gb
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.validateUsableSpace(SearchableSnapshotsConfigurationBean.java:133)
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.buildConfigurationPart(SearchableSnapshotsConfigurationBean.java:98)
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.buildConfigurationPart(SearchableSnapshotsConfigurationBean.java:62)
at org.graylog.datanode.configuration.OpensearchConfigurationService.lambda$get$0(OpensearchConfigurationService.java:109)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:110)
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:123)
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:67)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:65)
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104)
at java.base/java.lang.Thread.run(Unknown Source)
2025-11-24T15:46:10.919Z INFO [Datanode] SIGNAL received. Shutting down.
```
## Your Environment
* Graylog Version: 7.0.1
* Java Version: Bundled
* OpenSearch Version: Data Node 7.0.1
* MongoDB Version: 8.0.13
* Operating System: Ubuntu Server 22.04 LTS
* Browser version: Google Chrome Version 142.0.7444.176 (Official Build) (arm64)
## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?
Contributor guide
Assessment
This issue has not been assessed yet.