Graylog2 / Graylog2/graylog2-server

slow search & decreased retention in new graylog 3.2.2

Open
#7,599 5 comments 5 reactions 0 assignees View on GitHub
bug search triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

slow search & decreased retention on Graylog 3.2.2 version, because decreased amount of documents in index. Retention decreased from 13 days to 10 days.

We have two GRAYLOG clusters 5+10 with versions 3.0.2 and 3.2.2 side by side (blue green deployment) with the same settings:

Indices settings:
Shards: 10
Replicas: 1
Index Rotation strategy: Index Size
Max index size: 16GB
Index retention strategy: Delete
Max number of indices: 200

**Indices from version 3.2.2:**
graylog_270 Contains messages from 4 days ago up to 4 days ago (14.3GB / 29,612,635 messages) Show Details / Actions
graylog_269 Contains messages from 4 days ago up to 4 days ago (14.1GB / 29,234,745 messages) Show Details / Actions
graylog_268 Contains messages from 4 days ago up to 4 days ago (13.9GB / 28,817,818 messages) Show Details / Actions
graylog_267 Contains messages from 4 days ago up to 4 days ago (14.2GB / 31,456,249 messages) Show Details / Actions
graylog_266 Contains messages from 4 days ago up to 4 days ago (14.5GB / 29,905,324 messages) Show Details / Actions
AVG:**29,805,354** docs per index

**Indices from version 3.0.2:**
graylog_2713 Contains messages from 4 days ago up to 4 days ago (13.3GB / 34,044,967 messages) Show Details / Actions
graylog_2712 Contains messages from 4 days ago up to 4 days ago (12.9GB / 32,813,180 messages) Show Details / Actions
graylog_2711 Contains messages from 4 days ago up to 4 days ago (14.4GB / 38,896,557 messages) Show Details / Actions
graylog_2710 Contains messages from 4 days ago up to 4 days ago (13.0GB / 33,717,885 messages) Show Details / Actions
graylog_2709 Contains messages from 4 days ago up to 4 days ago (13.3GB / 34,285,394 messages) Show Details / Actions
**AVG: 34,746,196** docs per index

newer version of Graylog has fewer documents per index
**RATIO**: 29,805,354 / 34,746,196 *100 = **85.78%**

I was comparing messages in both GRAYLOG versions. I founded only field `gl2_accounted_message_size` in 3.2.2.
Is this flied responsible for that?

Also GUI searches takes more time in version 3.2.2.
| Compare searches in ms | v3.0.2 | v3.2.2 |
| -- | -- | -- |
| 5 min | 239 | 469 |
| 1 hour | 1 105 | 3036 |
| 8 hours | 2 292 | 9 094 |
| 1 day | 5 558 | 15 431 |
| 2 days | 12 343 | 34 426 |
| 5 days | 33 911 | time outed

**Slow GUI searches are big showstopper**

## Your Environment


* Graylog Version: 3.0.2 vs 3.2.2
* Elasticsearch Version: 6.8.6
* MongoDB Version: 4.0.13
* Operating System: Ubuntu 18.04
* Java Version: openjdk-8-jre-headless
* Browser version: Firefox 73.0.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.