opensearch-project / opensearch-project/data-prepper
TCP source plugin
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
I am in charge of collecting application logs. I have java applications that has logging.config written in logback.xml.
Loback.xml - part of sending logs to logstash looks like:
<appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>node1:port</destination>
<destination>node2:port</destination>
<destination>node3:port</destination>
<ssl>
<trustStore>
<location>file:/xxx/logstash.truststore</location>
<password>pw</password>
</trustStore>
</ssl>
<!-- encoder is required -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"...."}</customFields>
</encoder>
</appender>
With these settings on java APP server. The application sends data to logstash and Logstash is set to Server and on input has: TCP source plugin.
Can you add these feature to DataPrepper, so I can use DataPrepper instead of Logstash? I dont know another way to transmit logs to logstash from my app machine.
I am looking for alternative config that I have in Logstash OSS with OpenSearch Output Plugin:
input {
tcp {
mode => "server"
host => "IP"
port => "port"
ssl_enable => "true"
ssl_cert => "crt"
ssl_key => "key"
ssl_key_passphrase => "PW"
ssl_verify => "false"
ssl_cipher_suites => ['TLS_AES_256_GCM_SHA384', 'TLS_AES_128_GCM_SHA256', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256']
ssl_supported_protocols => ['TLSv1.2', 'TLSv1.3']
codec => "json_lines"
tags => "ssl_TCPinput"
}
}
filter {
if [LogType] == "TrxPersist" {
mutate { add_tag => "trx_log" }
}
else if [LogType] == "TrxPostProc" {
mutate { add_tag => "trx_time" }
}
if [appname] == "INT_EDDIE" {
mutate { add_field => { "[@metadata][target_index]" => "eddie-int" } }
}
}
output {
if [enviroment] == "integration" {
if [appname] == "INT_EDDIE" {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "%{[@metadata][target_index]}-temporary-%{+YYYY-MM-dd}"
manage_template => false
}
}
else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "trash-int-%{+YYYY.MM.dd}"
manage_template => false
}
}
}
else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "trash"
manage_template => false
}
}
}
I had conversation on forum before this report: https://forum.opensearch.org/t/logstash-conf-converter-to-data-prepper/12082/3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Data Prepper's existing source-plugin entry points and comparing them with the TCP source and TLS options described in the issue. Use the linked forum discussion for prior context. Done should mean a documented TCP server source can accept the described JSON-lines and TLS configuration and support the requested routing workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, networking, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100