logstash-filter-aggregate is not merging two CDR (logs) into one index in OpenSearch.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
**Logstash information**:
Please include the following information:
1. Logstash version logstash 7.17.6
2. Logstash installation source docker
3. How is Logstash being run docker
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
**JVM** (e.g. `java -version`):
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.
**OS version** (`uname -a` if on a Unix-like system):
Linux logstash-os-west2a-pvt.xyz.test.biz 5.10.209-198.858.amzn2.aarch64 https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1 SMP Tue Feb 13 18:46:45 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
**Description of the problem including expected versus actual behavior**:
I am having some logs in Opensearch name ecc-tdr & cdn-tdr , Both logs are there in one index name "ecc-cdn-combined4-prod-tdrs-2024.02"
Using logstash pipeline , I am taking input from "ecc-cdn-combined4-prod-tdrs-2024.02" index from Opensearch and using logstash-filter-aggregate plugin to merge both logs (Transaction call record ) in one merged index "ecc-cdn-merged"
Its not getting merged into one Logs (TDR) with all required field with ecc-tdr & cdn-tdr
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
`input {
opensearch {
hosts => ["https://vpc-production-bcd-us-es-xxxxxxx.us-west-2.es.amazonaws.com"]
user => "user"
password => "xxxxxxxx"
index => "ecc-cdn-combined4-prod-tdrs-2024.02"
#schedule => '*/2 * * * *'
ecs_compatibility => disabled
query => '{ "query": { "match_all": { } } }'
}
}
filter {
if [rcd_sent] == "logo" {
aggregate {
task_id => "%{transaction_id}"
code => "
map['ecc_customer'] = event.get('enterprise_billing_id')
map['type'] = event.get('type')
map['transaction_id'] = event.get('transaction_id')
map['ecc_req'] ||= event.get('raw_req_p')
map['ecc_resp'] ||= event.get('raw_resp_p')
map['ecc_request_time'] = event.get('request_time')
event.cancel()
"
map_action => "create"
}
}
if [rcd_sent] != "logo" {
aggregate {
task_id => "%{transaction_id}"
code => "
event.set('ecc_customer', map['ecc_customer'])
event.set('ecc_req', map['ecc_req'])
map['type'] = event.get('type')
event.cancel()
"
map_action => "create_or_update"
timeout_tags => ['_cdn_tdr_missing']
push_map_as_event_on_timeout => "true"
inactivity_timeout => 300
timeout_timestamp_field => "@timestamp"
timeout_task_id_field => "task_id"
timeout_code => "event.set('state', 'timeout')"
end_of_task => "true"
timeout => 18000000
}
}
}
output {
opensearch {
hosts => ["https://vpc-production-xxx-us-es-xxxxxxxxx.us-west-2.es.amazonaws.com:443"]
user => "user"
password => "xxxxxxx@"
index => "ecc-xx-merged10-prod-tdrs-2024.03"
}
}`
**Provide logs (if relevant)**:
No Failure Logs. However, expecting two logs to be combined in one logs having some fields where "transaction_id" is common
type=cdn-tdrs & type=ecc-tdrs values are there in both the logs
Contributor guide
Assessment
This issue has not been assessed yet.