Graylog2 / Graylog2/graylog2-server
allocation of `opendistro_security` shards fails without `skip_preflight_checks=true`
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
The allocation of `opendistro_security` shards works with `skip_preflight_checks=false` (which is the default)
## Current Behavior
The allocation of `opendistro_security` shards fails without `skip_preflight_checks=true`:
```text
index shard prirep state node
.opendistro_security 0 p STARTED datanode01
.opendistro_security 0 r UNASSIGNED ALLOCATION_FAILED
.opendistro_security 0 r UNASSIGNED ALLOCATION_FAILED
```
`_cluster/allocation/explain?pretty=true`:
```json
{
"index" : ".opendistro_security",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2025-12-05T12:36:05.246Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [bSQWXuVXEgiQWQJD3U3VHu]: failed recovery, failure RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} ([.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery))]; nested: RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery)]; nested: StringIndexOutOfBoundsException[Range [1, 0) out of bounds for length 0]; ",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "bSQWXuVXEgiQWQJD3U3VHu",
"node_name" : "datanode03",
"transport_address" : "192.0.2.6:9300",
"node_attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2025-12-05T12:36:05.246Z], failed_attempts[5], failed_nodes[[PKtmAYH1Ta-qHLihiYH6mA, bSQWXuVXEgiQWQJD3U3VHu]], delayed=false, details[failed shard on node [bSQWXuVXEgiQWQJD3U3VHu]: failed recovery, failure RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} ([.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery))]; nested: RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery)]; nested: StringIndexOutOfBoundsException[Range [1, 0) out of bounds for length 0]; ], allocation_status[no_attempt]]]"
}
]
},
{
"node_id" : "D9QLn4LhLNLOsCC6y8T2LA",
"node_name" : "datanode01",
"transport_address" : "192.0.2.4:9300",
"node_attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2025-12-05T12:36:05.246Z], failed_attempts[5], failed_nodes[[PKtmAYH1Ta-qHLihiYH6mA, bSQWXuVXEgiQWQJD3U3VHu]], delayed=false, details[failed shard on node [bSQWXuVXEgiQWQJD3U3VHu]: failed recovery, failure RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} ([.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery))]; nested: RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery)]; nested: StringIndexOutOfBoundsException[Range [1, 0) out of bounds for length 0]; ], allocation_status[no_attempt]]]"
},
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[.opendistro_security][0], node[D9QLn4LhLNLOsCC6y8T2LA], [P], s[STARTED], a[id=ntN8QFAVSGmZW8X1RJU5LA]]"
}
]
},
{
"node_id" : "PKtmAYH1Ta-qHLihiYH6mA",
"node_name" : "datanode02",
"transport_address" : "192.0.2.5:9300",
"node_attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2025-12-05T12:36:05.246Z], failed_attempts[5], failed_nodes[[PKtmAYH1Ta-qHLihiYH6mA, bSQWXuVXEgiQWQJD3U3VHu]], delayed=false, details[failed shard on node [bSQWXuVXEgiQWQJD3U3VHu]: failed recovery, failure RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} ([.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery))]; nested: RecoveryFailedException[[.opendistro_security][0]: Recovery failed from {datanode01}{D9QLn4LhLNLOsCC6y8T2LA}{m65nuU9ZADVDyjB7MTTGtT}{datanode01}{192.0.2.4:9300}{dimr}{shard_indexing_pressure_enabled=true} into {datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{IJd9yvbz3Q7AMtvPUrJEQ4}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (failed to clean after recovery)]; nested: StringIndexOutOfBoundsException[Range [1, 0) out of bounds for length 0]; ], allocation_status[no_attempt]]]"
}
]
}
]
}
```
We also saw some other logs which disappeared with `skip_preflight_checks=true`:
```
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in
MMapDirectory@/srv/opensearch_data/nodes/0/indices/M1t21UvRRw6SFsnwY30IQg/0/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@3711c71c: files: []
```
and
```
[2025-12-18T14:01:25,314][INFO ][o.o.c.c.FollowersChecker ] [datanode01] FollowerChecker{discoveryNode={datanode02}{PKtmAYH1Ta-qHLihiYH6mA}{XyhN70tNTOuk-o2j-JEjaw}{datanode02}{192.0.2.5:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2025-12-18T14:01:25,315][INFO ][o.o.c.c.FollowersChecker ] [datanode01] FollowerChecker{discoveryNode={datanode02}{PKtmAYH1Ta-qHLihiYH6mA}{XyhN70tNTOuk-o2j-JEjaw}{datanode02}{192.0.2.5:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2025-12-18T14:01:25,315][INFO ][o.o.c.c.FollowersChecker ] [datanode01] FollowerChecker{discoveryNode={datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{-vUjAnO_TjShZ_VSyoOBWA}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2025-12-18T14:01:25,315][INFO ][o.o.c.c.FollowersChecker ] [datanode01] FollowerChecker{discoveryNode={datanode03}{bSQWXuVXEgiQWQJD3U3VHu}{-vUjAnO_TjShZ_VSyoOBWA}{datanode03}{192.0.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2025-12-18T14:01:26,977][INFO ][o.o.n.Node ] [datanode01] stopped
[2025-12-18T14:01:26,978][INFO ][o.o.n.Node ] [datanode01] closing ...
[2025-12-18T14:01:26,978][ERROR][i.n.u.c.D.rejectedExecution] [datanode01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
```
## Steps to Reproduce (for bugs)
Unfortunately we are not sure. It randomly occurs during restarts of the Graylog DataNode.
## Your Environment
* Graylog Server Version: 7.0.2
* Java Version: `openjdk 21.0.9 2025-10-21 LTS`
* Graylog Datanode Version: 7.0.2
* MongoDB Version: 7.0.25
* Operating System: `Debian GNU/Linux 12 (bookworm)`
## Checklist
[ ] This issue fix need to be backported.
[ ] Does this issue have **security** implications?
Contributor guide
Assessment
This issue has not been assessed yet.