apache / apache/pinot

Error with Data size larger than 1M, will not write to zk. Data (first 1k)

Open
#7,704 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

This is not critical but shows some problems worth investigating.

Our QA cluster is having trouble with some accumulated data. It has 10k+ real time segments in place, data are not too much.

In logs we see something like below:
```
2021/11/04 20:31:34.166 ERROR [ZkClient] [HelixTaskExecutor-message_handle_thread] Data size larger than 1M, will not write to zk. Data (first 1k): {
"id" : "point_entry_REALTIME",
"simpleFields" : {
"BATCH_MESSAGE_MODE" : "false",
"BUCKET_SIZE" : "0",
"SESSION_ID" : "30069443a0581e1",
"STATE_MODEL_DEF" : "SegmentOnlineOfflineStateModel",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
},
"mapFields" : {
"point_entry__0__0__20211030T0056Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__100__20211102T0746Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__101__20211102T0817Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__102__20211102T0909Z" : {
"CURRENT_STATE" : "OFFLINE"
},
"point_entry__0__103__20211102T0946Z" : {
"CURRENT_STATE" : "ONLINE",
"END_TIME" : "1636056441791",
"INFO
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ZkClient write path associated with the logged size-limit error and examine how the accumulated real-time segment data reaches it. Reproduce the issue with data larger than 1M and document a verified outcome for handling or reporting that condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.