opensearch-project / opensearch-project/data-prepper

[BUG] STDOUT not respecting Log4J configuration

Open
#2,116 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
I've updated my log4j configuration and it stopped to save logs o file system, so, it worked well.
I've fixed all log levels to errors but data-prepper continues printing the content of the logs received.

To Reproduce
1: Updated log4j configutation at '/usr/share/data-prepper/config/log4j2-rolling.properties' with:

status = error
dest = err
name = PropertiesConfig

property.filename = log/data-prepper/data-prepper.log

appender.console.type = Console
appender.console.level = error
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n

rootLogger.level = error
rootLogger.appenderRef.stdout.ref = STDOUT

logger.pipeline.name = org.opensearch.dataprepper.pipeline
logger.pipeline.level = error

logger.parser.name = org.opensearch.dataprepper.parser
logger.parser.level = error

logger.plugins.name = org.opensearch.dataprepper.plugins
logger.plugins.level = error
  1. I am currently using the default pipeline for open telemetry use case (/usr/share/data-prepper/pipelines/pipelines.yaml):
entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: [ "https://opensearch.stag.bolttechbroker.net" ]
        username: "edirect"
        password: "123@PenguinsCanFly;)"
        index_type: "trace-analytics-raw"
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://opensearch.stag.bolttechbroker.net" ]
        username: "edirect"
        password: "123@PenguinsCanFly;)"
        index_type: "trace-analytics-service-map"

Expected behavior
I dont want to have all requests logged to my console, I just want to have errors logged to it.

Screenshots
image

Environment (please complete the following information):

  • OS: Debian
  • Version: latest

Additional context
I am currently using the lates version of data-prepper

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with /usr/share/data-prepper/config/log4j2-rolling.properties and verify that Data Prepper loads this configuration when using /usr/share/data-prepper/pipelines/pipelines.yaml. Reproduce the default OpenTelemetry pipeline with the shown logger levels, then trace why received request content is still printed. Done means console output is limited to errors while the pipeline continues processing normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.