apache / apache/beam

Reduce log noise coming from File sink

Open
#18,698 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

After switching to Beam 2.2 I noticed that File sink related classes generate lots of lines likes this:
```

2018-01-16 01:37:37,080 INFO org.apache.beam.sdk.io.FileBasedSink - No output files to write.
2018-01-16
01:37:37,104 INFO org.apache.beam.sdk.io.WriteFiles - Will finalize 0 files
```

 

I did some counts and it looks like this lines account for 82% of all lines in the log yet not giving much information.

I am happy to raise a PR to make "No output files to write." and "Will finalize {} files" logged at DEBUG level.

Imported from Jira [BEAM-3488](https://issues.apache.org/jira/browse/BEAM-3488). Original Jira may contain additional context.
Reported by: pawelbartoszek.

Contributor guide

Open the contributing guide

Research direction

Locate the logging statements in the FileBasedSink and WriteFiles classes that emit "No output files to write." and "Will finalize {} files". Confirm the desired behavior is for both messages to use DEBUG logging, then verify that the File sink no longer produces these messages at INFO level.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.