opensearch-project / opensearch-project/data-prepper
[BUG] STDOUT not respecting Log4J configuration
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
I've updated my log4j configuration and it stopped to save logs o file system, so, it worked well.
I've fixed all log levels to errors but data-prepper continues printing the content of the logs received.
To Reproduce
1: Updated log4j configutation at '/usr/share/data-prepper/config/log4j2-rolling.properties' with:
status = error
dest = err
name = PropertiesConfig
property.filename = log/data-prepper/data-prepper.log
appender.console.type = Console
appender.console.level = error
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n
rootLogger.level = error
rootLogger.appenderRef.stdout.ref = STDOUT
logger.pipeline.name = org.opensearch.dataprepper.pipeline
logger.pipeline.level = error
logger.parser.name = org.opensearch.dataprepper.parser
logger.parser.level = error
logger.plugins.name = org.opensearch.dataprepper.plugins
logger.plugins.level = error
- I am currently using the default pipeline for open telemetry use case (/usr/share/data-prepper/pipelines/pipelines.yaml):
entry-pipeline:
delay: "100"
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
sink:
- opensearch:
hosts: [ "https://opensearch.stag.bolttechbroker.net" ]
username: "edirect"
password: "123@PenguinsCanFly;)"
index_type: "trace-analytics-raw"
service-map-pipeline:
delay: "100"
source:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
sink:
- opensearch:
hosts: [ "https://opensearch.stag.bolttechbroker.net" ]
username: "edirect"
password: "123@PenguinsCanFly;)"
index_type: "trace-analytics-service-map"
Expected behavior
I dont want to have all requests logged to my console, I just want to have errors logged to it.
Screenshots

Environment (please complete the following information):
- OS: Debian
- Version: latest
Additional context
I am currently using the lates version of data-prepper
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with /usr/share/data-prepper/config/log4j2-rolling.properties and verify that Data Prepper loads this configuration when using /usr/share/data-prepper/pipelines/pipelines.yaml. Reproduce the default OpenTelemetry pipeline with the shown logger levels, then trace why received request content is still printed. Done means console output is limited to errors while the pipeline continues processing normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100