elastic / elastic/logstash

[input elasticsearch] logstash does not execute certain queries correctly.

Open
#15,552 0 comments 0 reactions 0 assignees View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Hi there
i do not understand the behavior of logstash.
Although the field `is_read` exists, a successful query is still performed and an e-mail is sent.

```
input {
elasticsearch {
hosts => "https://elastic01:9200"
ssl => true
ca_file => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "password"
index => "test-index-default"
query => {"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}
schedule => "* * * * *"
size => 500
scroll => "1m"
docinfo => true
docinfo_target => "[@metadata][doc]"
}
}
filter {
json {
source => "message"
}
mutate {
replace => {"is_read"=> "true"}
}
}
output {
stdout {}
email {
to => "testuser@group.com"
address => "test.smtp"
subject => "Filebeat"
body => 'Test Message'
port => "25"
#username => "xxx@gmail.com"
#password => "****"
#use_tls => true
}
elasticsearch {
hosts => "https://elastic01:9200"
ssl => true
cacert => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "password"

retry_on_conflict => 5
index => "test-index-default"
document_type => "%{[@metadata][doc][_type]}"
document_id => "%{[@metadata][doc][_id]}"
action => "update"
}
}
```
The query itself can be carried out successfully in Kibana.
Hence the question, why is this query successful?
`{"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}`
Although the field exists and therefore a mail is still sent every minute.

**Logstash information**:

Please include the following information:

1. Logstash version: **8.11.0**
2. Logstash installation source: **RPM**
3. How is Logstash being run: **systemd**

**Plugins installed**: (`bin/logstash-plugin list --verbose`)
```
Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.0)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.3)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.0)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.7)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.0)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.7.1)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.3)
logstash-input-elasticsearch (4.18.0)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.5)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.7.3)
logstash-input-http_poller (5.5.0)
logstash-input-imap (3.2.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search (2.2.1)
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.6)
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.2)
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash (0.0.5)
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv (3.0.9)
logstash-output-elasticsearch (11.19.0)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)
```

**JVM** (e.g. `java -version`):
```
Using bundled JDK: /usr/share/logstash/jdk
```
**OS version** (`uname -a` if on a Unix-like system):
```
Linux elastic01 5.14.21-150500.55.31-default #1 SMP PREEMPT_DYNAMIC Wed Oct 4 16:52:05 UTC 2023 (5dc23e0) x86_64 x86_64 x86_64 GNU/Linux
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Logstash 8.11.0 pipeline using the elasticsearch input query, mutate filter, email output, and Elasticsearch update output shown in the issue. Compare the scheduled query results with the fields reaching the email output; done means the cause of repeated emails for documents with is_read is identified and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.