opensearch-project / opensearch-project/data-prepper

TCP source plugin

Open
#2,162 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement plugin - source
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

I am in charge of collecting application logs. I have java applications that has logging.config written in logback.xml.
Loback.xml - part of sending logs to logstash looks like:

    <appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
                 <destination>node1:port</destination>
                 <destination>node2:port</destination>
                 <destination>node3:port</destination>
                 <ssl>
                         <trustStore>
                                <location>file:/xxx/logstash.truststore</location>
                                <password>pw</password>
                         </trustStore>
                 </ssl>
                <!-- encoder is required -->
                <encoder class="net.logstash.logback.encoder.LogstashEncoder">
                <customFields>{"...."}</customFields>
                </encoder>
        </appender>

With these settings on java APP server. The application sends data to logstash and Logstash is set to Server and on input has: TCP source plugin.

Can you add these feature to DataPrepper, so I can use DataPrepper instead of Logstash? I dont know another way to transmit logs to logstash from my app machine.

I am looking for alternative config that I have in Logstash OSS with OpenSearch Output Plugin:

input {
  tcp {
    mode => "server"
    host => "IP"
    port => "port"
    ssl_enable => "true"
    ssl_cert => "crt"
    ssl_key => "key"
    ssl_key_passphrase => "PW"
    ssl_verify => "false"
    ssl_cipher_suites => ['TLS_AES_256_GCM_SHA384', 'TLS_AES_128_GCM_SHA256', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256']
    ssl_supported_protocols => ['TLSv1.2', 'TLSv1.3']
    codec => "json_lines"
    tags => "ssl_TCPinput"
  }
}

filter {
if [LogType] == "TrxPersist" {
        mutate { add_tag => "trx_log" }
}
else if [LogType] == "TrxPostProc" {
  mutate { add_tag => "trx_time" }
 }

if [appname] == "INT_EDDIE" {
mutate { add_field => { "[@metadata][target_index]" => "eddie-int" } }
}
}

output {
if [enviroment] == "integration" {
if [appname] == "INT_EDDIE" {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "%{[@metadata][target_index]}-temporary-%{+YYYY-MM-dd}"
manage_template => false
}
}

else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "trash-int-%{+YYYY.MM.dd}"
manage_template => false
}
}
}

else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "user"
password => "pw"
index => "trash"
manage_template => false
}
}
}

I had conversation on forum before this report: https://forum.opensearch.org/t/logstash-conf-converter-to-data-prepper/12082/3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Data Prepper's existing source-plugin entry points and comparing them with the TCP source and TLS options described in the issue. Use the linked forum discussion for prior context. Done should mean a documented TCP server source can accept the described JSON-lines and TLS configuration and support the requested routing workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.