apache / apache/accumulo

Reduce merge info logging.

Open
#2,788 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

The current merge info that is printed to the logs is spammy, but may require significant changes to reduce the logging while still providing some useful information to monitor / trouble shooting merge operations. The core issue is that each time the next merge state is recalculated the merge overlap information is logged.

One way may be to print the overlaps on creation and on completion, but suppress or summarize the intermediate states.

The current logging (for a small merge) for each merge calculation looks like:
```
2022-06-28T14:21:58,034 [manager.Manager] DEBUG: Creating merge op: 3 from startRow: -inf to endRow: +inf
2022-06-28T14:21:58,175 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<

```

Click to expand for full merge operation example

```
2022-06-28T14:21:58,034 [manager.Manager] DEBUG: Creating merge op: 3 from startRow: -inf to endRow: +inf
2022-06-28T14:21:58,175 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently STARTED isDelete : false
2022-06-28T14:21:58,288 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,290 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,290 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,291 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,292 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:21:58,908 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,911 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,911 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,912 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,912 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:21:59,021 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:59,021 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_OFFLINE isDelete : false
2022-06-28T14:21:59,163 [state.MergeInfo] DEBUG: mergeInfo 3<< overlaps: 3<<
2022-06-28T14:21:59,166 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently COMPLETE isDelete : false
2022-06-28T14:21:59,178 [merge.TableRangeOpWait] INFO : removing merge information No Merge in progress

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the merge logging shown from state.MergeInfo, state.MergeStats, manager.Manager, and merge.TableRangeOpWait. Review when overlap information is emitted during merge-state recalculation, then define a focused logging change that preserves useful creation and completion information while suppressing or summarizing intermediate output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.