Graylog2 / Graylog2/graylog2-server

Restarting Individual Docker Datanode Fails

Open
#25,435 1 comment 0 reactions 0 assignees View on GitHub
bug test-day triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

**I came across this issue testing datanode in Docker, so it might be a docker image issue rather than a datanode code issue.**

## Expected Behavior

After stopping an individual datanode from the System > Cluster Configuration > Data Nodes accordion, restarting the node should add it back as AVAILABLE
## Current Behavior

After stopping an individual datanode from the System > Cluster Configuration > Data Nodes accordion, restarting the node leaves it in an UNAVAILABLE state with the following error message in the logs

```
2026-03-24T20:54:05.047Z INFO [ClusterNodeStateTracer] Updating cluster node 2efc2fde-443b-4844-bcdf-1415ccfa117d from UNAVAILABLE to STARTING (reason: PROCESS_STARTED)
2026-03-24T20:54:09.791Z WARN [OpensearchProcessImpl] WARNING: A restricted method in java.lang.System has been called
2026-03-24T20:54:09.791Z WARN [OpensearchProcessImpl] WARNING: java.lang.System::load has been called by com.sun.jna.Native in an unnamed module (file:/usr/share/graylog-datanode/dist/opensearch-3.5.0-linux-x64/lib/jna-5.16.0.jar)
2026-03-24T20:54:09.792Z WARN [OpensearchProcessImpl] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
2026-03-24T20:54:09.792Z WARN [OpensearchProcessImpl] WARNING: Restricted methods will be blocked in a future release unless native access is enabled
2026-03-24T20:54:09.792Z WARN [OpensearchProcessImpl]
2026-03-24T20:54:10.144Z INFO [OpensearchProcessImpl] 2026-03-24T20:54:10.141782843Z main ERROR fatal error in thread [main], exiting
2026-03-24T20:54:10.144Z INFO [OpensearchProcessImpl] java.lang.NoClassDefFoundError: org/opensearch/javaagent/bootstrap/AgentPolicy$AnyCanExit
2026-03-24T20:54:10.144Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Security.configure(Security.java:165)
2026-03-24T20:54:10.144Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:245)
2026-03-24T20:54:10.145Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
2026-03-24T20:54:10.145Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
2026-03-24T20:54:10.145Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] fatal error in thread [main], exiting
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] Caused by: java.lang.ClassNotFoundException: org.opensearch.javaagent.bootstrap.AgentPolicy$AnyCanExit
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] java.lang.NoClassDefFoundError: org/opensearch/javaagent/bootstrap/AgentPolicy$AnyCanExit
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Security.configure(Security.java:165)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:245)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl] ... 10 more
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
2026-03-24T20:54:10.146Z INFO [OpensearchProcessImpl]
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] Caused by: java.lang.ClassNotFoundException: org.opensearch.javaagent.bootstrap.AgentPolicy$AnyCanExit
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
2026-03-24T20:54:10.146Z WARN [OpensearchProcessImpl] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
2026-03-24T20:54:10.147Z WARN [OpensearchProcessImpl] ... 10 more
2026-03-24T20:54:10.284Z WARN [OpensearchProcessImpl] Opensearch process failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:408)
at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:322)
at java.base/java.lang.Thread.run(Unknown Source)
```
## Possible Solution

## Steps to Reproduce (for bugs)

1. Deploy multiple datanode nodes via docker
2. Stop and attempt to restart one of the nodes

## Context

test day failure

## Your Environment

* Graylog Version: 7.1.0-alpha.3-1
* Datanode Image: 7.1.0-alpha.3-1

## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.