Graylog2 / Graylog2/graylog2-server

Data Node Fails to start if available disk space is less than `node_search_cache_size`, despite used space being consumed by the cache itself

Open
#24,340 1 comment 0 reactions 1 assignee Claimed by @todvora View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Data Node Fails to start if available disk space is less than `node_search_cache_size`, despite used space being consumed by the cache itself.

The size of the cache should be subtracted from the used space when checking for available space.

The current logic may require space for the 2x the cache size.

## Expected Behavior

Graylog Data Node can start if the available space less the existing cache size is large enough to fit the configured cache size (`node_search_cache_size`).

## Current Behavior

Graylog Data Node cannot start if there the amount of free space is less than `node_search_cache_size` despite the existing cache size consuming the same amount of space as `node_search_cache_size`.

## Possible Solution

Account for the existing cache size when calculating available space.

## Steps to Reproduce (for bugs)

1.
2.
3.
4.

## Context

Ran into this when my Data Node used exclusively for warm tier failed to start:

```
2025-11-24T15:46:10.916Z ERROR [DatanodeBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:765)
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:586)
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:300)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.startCommand(DatanodeBootstrap.java:138)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:382)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
Caused by: org.graylog.datanode.configuration.OpensearchConfigurationException: There is not enough usable space for the node search cache. Your system has only 36gb available.
Either decrease node_search_cache_size configuration or make sure that datanode has enough free disk space.
Data directory: /var/lib/opensearch, current node_search_cache_size: 50gb
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.validateUsableSpace(SearchableSnapshotsConfigurationBean.java:133)
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.buildConfigurationPart(SearchableSnapshotsConfigurationBean.java:98)
at org.graylog.datanode.opensearch.configuration.beans.impl.SearchableSnapshotsConfigurationBean.buildConfigurationPart(SearchableSnapshotsConfigurationBean.java:62)
at org.graylog.datanode.configuration.OpensearchConfigurationService.lambda$get$0(OpensearchConfigurationService.java:109)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:110)
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:123)
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:67)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:65)
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104)
at java.base/java.lang.Thread.run(Unknown Source)
2025-11-24T15:46:10.919Z INFO [Datanode] SIGNAL received. Shutting down.
```

## Your Environment

* Graylog Version: 7.0.1
* Java Version: Bundled
* OpenSearch Version: Data Node 7.0.1
* MongoDB Version: 8.0.13
* Operating System: Ubuntu Server 22.04 LTS
* Browser version: Google Chrome Version 142.0.7444.176 (Official Build) (arm64)

## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.