Graylog2 / Graylog2/graylog2-server

IndexFieldTypePollerPeriodical throws occasionally ResponseException

Open
#19,227 1 comment 0 reactions 1 assignee Claimed by @dennisoelkers View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

No errors in the logs.

## Current Behavior

This error shows occasionally in the logs. It refers to the Graylog Events index set.

```
2024-05-02T15:18:01.048+03:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: Couldn't get index list for index set <5d89b76d0a25285d26afd396>
at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:182) ~[?:?]
at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:133) ~[?:?]
at org.graylog.storage.opensearch2.cat.CatApi.perform(CatApi.java:105) ~[?:?]
at org.graylog.storage.opensearch2.cat.CatApi.requestIndices(CatApi.java:101) ~[?:?]
at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:76) ~[?:?]
at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:71) ~[?:?]
at org.graylog.storage.opensearch2.IndicesAdapterOS2.indices(IndicesAdapterOS2.java:414) ~[?:?]
at org.graylog2.indexer.indices.Indices.getIndices(Indices.java:283) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePoller.poll(IndexFieldTypePoller.java:68) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$refreshFieldTypes$2(IndexFieldTypePollerPeriodical.java:155) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Unknown Source) ~[?:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.refreshFieldTypes(IndexFieldTypePollerPeriodical.java:144) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.doRun(IndexFieldTypePollerPeriodical.java:131) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [GET], host [https://data1:9200], URI [/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"},"status":400}
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:375) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:345) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:320) ~[?:?]
at org.graylog.storage.opensearch2.cat.CatApi.lambda$perform$4(CatApi.java:108) ~[?:?]
at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:131) ~[?:?]
... 18 more
```

## Possible Solution

No idea. The storage driver for Opensearch does something wrong?

## Steps to Reproduce (for bugs)

It happens on its own.

The API from Opensearch seems to give proper information:
```
$ curl -k -X GET "https://admin:passwowrd@data1:9200/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all&pretty"
[
{
"index" : "gl-events_10",
"status" : "open"
},
{
"index" : "gl-events_11",
"status" : "open"
},
{
"index" : "gl-events_12",
"status" : "open"
},
{
"index" : "gl-events_3",
"status" : "open"
},
{
"index" : "gl-events_4",
"status" : "open"
},
{
"index" : "gl-events_5",
"status" : "open"
},
{
"index" : "gl-events_6",
"status" : "open"
},
{
"index" : "gl-events_7",
"status" : "open"
},
{
"index" : "gl-events_8",
"status" : "open"
},
{
"index" : "gl-events_9",
"status" : "open"
}
]

```

## Context

I have no idea what this might cause.

## Your Environment

* Graylog Version: 5.2.6
* Java Version: Bundled 17.x
* OpenSearch Version: 2.13
* MongoDB Version: 5.0.26
* Operating System: RHEL8
* Browser version: n/a

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.