elastic / elastic/integrations

[Logstash] Always reports 0 for in memory queue

Open
#14,121 1 comment 0 reactions 0 assignees View on GitHub
Integration:logstash needs:triage
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Logstash [logstash]

### Dataset Name

logstash.node

### Integration Version

2.7.1

### Agent Version

8.18.

### Agent Output Type

elasticsearch

### Elasticsearch Version

8.18

### OS Version and Architecture

docker

### Software/API Version

_No response_

### Error Message

The field `logstash.node.stats.pipelines.queue.events_count` is always 0 for in memory queues, it is only populated when you use persistent disk. In general all `queue` metrics are empty for in memory queue:

```
"queue": {
"events_count": 0,
"max_queue_size_in_bytes": 0,
"queue_size_in_bytes": 0,
"type": "memory"
},
```

Taken from https://www.elastic.co/docs/reference/integrations/logstash/#node-stats

### Event Original

```
{
"ephemeral_id": "7114cd7d-8d91-4afc-a986-32487c3edcbe",
"events": {
"duration_in_millis": 191,
"filtered": 91,
"in": 95,
"out": 91,
"queue_push_duration_in_millis": 4
},
"hash": "0542fa70daa36dc3e858ea099f125cc8c9e451ebbfe8ea8867e52f9764da0a35",
"id": "pipeline-with-memory-queue",
"queue": {
"events_count": 0,
"max_queue_size_in_bytes": 0,
"queue_size_in_bytes": 0,
"type": "memory"
},
"reloads": {
"failures": 0,
"successes": 0
},
```

Directly form the documentation https://www.elastic.co/docs/reference/integrations/logstash/#node-stats

### What did you do?

Use the new logstash monitoring

### What did you see?

Missing `logstash.node.stats.pipelines.queue.events_count` apropriate value for that, representing the actual queue size.

### What did you expect to see?

The actual amount of docs in the in memory queue

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.