[input elasticsearch] logstash does not execute certain queries correctly.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Hi there
i do not understand the behavior of logstash.
Although the field `is_read` exists, a successful query is still performed and an e-mail is sent.
```
input {
elasticsearch {
hosts => "https://elastic01:9200"
ssl => true
ca_file => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "password"
index => "test-index-default"
query => {"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}
schedule => "* * * * *"
size => 500
scroll => "1m"
docinfo => true
docinfo_target => "[@metadata][doc]"
}
}
filter {
json {
source => "message"
}
mutate {
replace => {"is_read"=> "true"}
}
}
output {
stdout {}
email {
to => "testuser@group.com"
address => "test.smtp"
subject => "Filebeat"
body => 'Test Message'
port => "25"
#username => "xxx@gmail.com"
#password => "****"
#use_tls => true
}
elasticsearch {
hosts => "https://elastic01:9200"
ssl => true
cacert => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "password"
retry_on_conflict => 5
index => "test-index-default"
document_type => "%{[@metadata][doc][_type]}"
document_id => "%{[@metadata][doc][_id]}"
action => "update"
}
}
```
The query itself can be carried out successfully in Kibana.
Hence the question, why is this query successful?
`{"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}`
Although the field exists and therefore a mail is still sent every minute.
**Logstash information**:
Please include the following information:
1. Logstash version: **8.11.0**
2. Logstash installation source: **RPM**
3. How is Logstash being run: **systemd**
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
```
Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.0)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.3)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.0)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.7)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.0)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.7.1)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.3)
logstash-input-elasticsearch (4.18.0)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.5)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.7.3)
logstash-input-http_poller (5.5.0)
logstash-input-imap (3.2.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search (2.2.1)
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.6)
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.2)
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash (0.0.5)
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv (3.0.9)
logstash-output-elasticsearch (11.19.0)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)
```
**JVM** (e.g. `java -version`):
```
Using bundled JDK: /usr/share/logstash/jdk
```
**OS version** (`uname -a` if on a Unix-like system):
```
Linux elastic01 5.14.21-150500.55.31-default #1 SMP PREEMPT_DYNAMIC Wed Oct 4 16:52:05 UTC 2023 (5dc23e0) x86_64 x86_64 x86_64 GNU/Linux
```
Contributor guide
Research direction
Start by reproducing the Logstash 8.11.0 pipeline using the elasticsearch input query, mutate filter, email output, and Elasticsearch update output shown in the issue. Compare the scheduled query results with the fields reaching the email output; done means the cause of repeated emails for documents with is_read is identified and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100