apache / apache/hudi

Reduce log volume by changing per-file/per-block logs to DEBUG level

Open
#14,356 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
Change verbose INFO logs to DEBUG level in the following classes:

1. `BaseHoodieLogRecordReader.java`:
- Change per-file logs ("Scanning log file") to DEBUG
- Change per-block logs ("Reading a data block", "Reading a delete block", "Reading a command block") to DEBUG
- Change rollback logs to DEBUG
- Add a single summary INFO log in the finally block that reports aggregate statistics (total log files, total blocks, total rollbacks, total corrupt blocks)

2. `SpillableMapBasedFileSystemView.java`:
- Add a single INFO log in the constructor with all memory config values
- Change individual map creation logs ("Creating Partition To File groups map", "Creating Pending Compaction map", etc.) to DEBUG

**Why this task is needed:**
Currently, these classes log at INFO level for every log file scanned and every block read. In tables with many partitions or log files, this generates excessive log volume that:

1. Makes it difficult to find important logs in production environments
2. Increases storage costs for log aggregation systems
3. Can impact performance due to I/O overhead

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading BaseHoodieLogRecordReader.java and SpillableMapBasedFileSystemView.java, locating the per-file, per-block, rollback, and map-creation INFO logs. Change the verbose messages to DEBUG, retain aggregate and memory-configuration summaries at INFO, and verify that the requested statistics and configuration values are represented in those summaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.