apache / apache/pinot

Improve segment-level DebugInfo

Open
#7,910 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

This issue was discussed partially in https://github.com/apache/pinot/pull/7896.

Originally the issue we discovered was that there were null stacktrace DebugInfo returned from the DebugAPI, (see below). We found out that 2 of the place stacktrace were swallowed.

While we address this issue by making the error explicitly thrown. we realized that the control flow were not as expected. There are several concerns:
@sajjad-moradi : If we throw exception in `buildSegmentAndRepalce` without correctly capture and change state, then the segment goes to error state in external view upon receiving helix transition message consuming -> online. see https://github.com/apache/pinot/pull/7896#discussion_r768340557
@mcvsubbu : status.ERROR should not be used when a segment creation was discarded because this another host might've been able to build it correctly. see: https://github.com/apache/pinot/pull/7896#discussion_r769141107; and when detecting a recoverable error we should not include that information in the debug info page in the first place. see: https://github.com/apache/pinot/pull/7896#discussion_r769160523

there's another question I notice from myself and @Jackie-Jiang is that the segment errors were captured and put in the debug API in almost all methods (public/protected/private), can we clean up the logic by creating a rule so that:
only external facing methods should capture and log debug info; and all internal methods should only bubble up the exception for external facing methods to capture and log with more contextual information?

Please discuss.

Click to expand code block!

```
[
{
"tableName": "testTable_REALTIME",
"numSegments": 1,
"numServers": 1,
"numBrokers": 1,
"segmentDebugInfos": [
{
"segmentName": "testTable__0__0__20211210T0838Z",
"serverState": {
"Server_localhost_8098": {
"idealState": "CONSUMING",
"externalView": "CONSUMING",
"segmentSize": "0 bytes",
"consumerInfo": {
"segmentName": "testTable__0__0__20211210T0838Z",
"consumerState": "NOT_CONSUMING",
"lastConsumedTimestamp": 1639125751331,
"partitionToOffsetMap": {
"0": "325618872"
}
},
"errorInfo": {
"timestamp": "2021-12-10 08:42:35 UTC",
"errorMessage": "Could not build segment",
"stackTrace": null
}
}
}
}
],
"serverDebugInfos": [],
"brokerDebugInfos": [
{
"brokerName": "Broker_localhost_8099",
"idealState": "ONLINE",
"externalView": "ONLINE"
}
],
"tableSize": {
"reportedSize": "0 bytes",
"estimatedSize": "0 bytes"
},
"ingestionStatus": {
"ingestionState": "UNHEALTHY",
"errorMessage": "Segment: testTable__0__0__20211210T0838Z is not being consumed on server: Server_localhost_8098"
}
}
]
```

Contributor guide

Open the contributing guide

Research direction

Start with the DebugAPI entry points and the buildSegmentAndRepalce flow, then read the discussion in PR 7896. Trace where segment exceptions are captured, states are changed, and debug information is assembled. Done means the capture rule is agreed for external versus internal methods, stack traces are not swallowed, and discarded builds are distinguished from unrecoverable errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems, observability
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.