apache / apache/pinot

Errors after Pinot server ran out of max map count

Open
#6,449 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

My EKS node running pinot server ran out of map areas and I bumped it to 262144 from default.
When the error came Pinot stopped consuming segments. I updated the max map in os and restarted the server. After starting Pinot server
the realtime table was showing Bad status. However it recovered eventually. I am seeing following message after that and
wondering is this due to some hidden problem somewhere in Pinot server ?
```
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
```
```
my-pinot-server-1 server 2021/01/16 18:50:51.938 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__162__20210112T1932Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__163__20210112T1935Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__164__20210112T1937Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__165__20210112T1940Z
my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__166__20210112T1944Z
my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__167__20210112T1947Z
my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__168__20210112T1950Z
my-pinot-server-1 server 2021/01/16 18:50:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__169__20210112T1953Z
n config.
```
```
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.autotune.initialRows' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'streamType' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.zk.broker.url' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
```
```
my-pinot-server-1 server 2021-01-16T19:01:25.952+0000: 672.343: Total time for which application threads were stopped: 0.0003745 seconds, Stopping threads took: 0.0000540 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.158+0000: 672.549: Application time: 0.2053891 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.159+0000: 672.550: Total time for which application threads were stopped: 0.0015166 seconds, Stopping threads took: 0.0000398 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.227+0000: 672.618: Application time: 0.0673477 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.228+0000: 672.619: Total time for which application threads were stopped: 0.0015950 seconds, Stopping threads took: 0.0000456 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.242+0000: 672.633: Application time: 0.0142821 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.243+0000: 672.634: Total time for which application threads were stopped: 0.0003874 seconds, Stopping threads took: 0.0000330 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Application time: 0.0067833 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Total time for which application threads were stopped: 0.0003493 seconds, Stopping threads took: 0.0000236 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.251+0000: 672.642: Application time: 0.0010896 seconds
my-pinot-server-1 server 2021-01-16T19:01:26.253+0000: 672.643: Total time for which application threads were stopped: 0.0014404 seconds, Stopping threads took: 0.0000105 seconds
my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Application time: 1.0000987 seconds
my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Total time for which application threads were stopped: 0.0004062 seconds, Stopping threads took: 0.0000532 seconds
my-pinot-server-1 server 2021-01-16T19:01:30.253+0000: 676.644: Application time: 3.0002441 seconds
my-pinot-server-1 server 2021-01-16T19:01:30.254+0000: 676.645: Total time for which application threads were stopped: 0.0003946 seconds, Stopping threads took: 0.0000391 seconds
my-pinot-server-1 server 2021-01-16T19:02:17.596+0000: 723.987: Application time: 47.3426464 seconds
my-pinot-server-1 server 2021-01-16T19:02:17.597+0000: 723.988: Total time for which application threads were stopped: 0.0004295 seconds, Stopping threads took: 0.0000559 seconds
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the logged TopStateHandoffReportStage, RealtimeTableDataManager, and ConsumerConfig entry points, then correlate them with the max map count restart and the realtime table's Bad status. Review the reported deletion failures and unknown Kafka configuration warnings; done means determining whether they indicate a Pinot defect or expected recovery behavior and documenting the conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, kubernetes
Domain
cloud, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.