apache / apache/hudi

Add SamplingLogger utility for reducing log volume while maintaining observability

Open
#14,353 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

Add a SamplingLogger utility class that logs at INFO level once every N times, otherwise at DEBUG level. Apply it to high-frequency log statements in AbstractTableFileSystemView:
- "Building file system view for partition(s)" - DEBUG for metadata tables, INFO every 10th time for data tables
- "Took X ms to read Y instants, Z replaced file groups" - INFO every 5th time

**Why this task is needed:**

In high-throughput scenarios with many partitions, these log statements generate excessive volume at INFO level, making it difficult to identify important events and increasing log storage costs. Sampling maintains periodic visibility for monitoring while reducing log volume by 80-90%.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate AbstractTableFileSystemView and identify the two high-frequency log statements described in the issue. Then find the appropriate location for a SamplingLogger utility and inspect nearby logging conventions. Done means the specified messages use the requested INFO/DEBUG sampling rates while preserving periodic visibility; no test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability-sre
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.