Reduce log volume by changing per-file/per-block logs to DEBUG level
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
Change verbose INFO logs to DEBUG level in the following classes:
1. `BaseHoodieLogRecordReader.java`:
- Change per-file logs ("Scanning log file") to DEBUG
- Change per-block logs ("Reading a data block", "Reading a delete block", "Reading a command block") to DEBUG
- Change rollback logs to DEBUG
- Add a single summary INFO log in the finally block that reports aggregate statistics (total log files, total blocks, total rollbacks, total corrupt blocks)
2. `SpillableMapBasedFileSystemView.java`:
- Add a single INFO log in the constructor with all memory config values
- Change individual map creation logs ("Creating Partition To File groups map", "Creating Pending Compaction map", etc.) to DEBUG
**Why this task is needed:**
Currently, these classes log at INFO level for every log file scanned and every block read. In tables with many partitions or log files, this generates excessive log volume that:
1. Makes it difficult to find important logs in production environments
2. Increases storage costs for log aggregation systems
3. Can impact performance due to I/O overhead
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading BaseHoodieLogRecordReader.java and SpillableMapBasedFileSystemView.java, locating the per-file, per-block, rollback, and map-creation INFO logs. Change the verbose messages to DEBUG, retain aggregate and memory-configuration summaries at INFO, and verify that the requested statistics and configuration values are represented in those summaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100