apache / apache/pinot

Pinot stop consuming from Kinesis Data Stream

Open
#9,869 16 comments 0 reactions 0 assignees View on GitHub
ingestion kinesis real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Pinot realtime ingestion from kinesis data stream works as expected for sometime but eventually stops consuming. While checking server logs, `connection refused` error is shown. Is this due to the issue mentioned in this PR #9863.
```
2022/11/29 16:10:58.538 INFO [ControllerLeaderLocator] [testtable__0__0__20221129T1419Z] Millis since last controller cache value invalidate 18567 is less than allowed frequency 30000. Skipping invalidate.
2022/11/29 16:10:58.538 INFO [LLRealtimeSegmentDataManager_testtable__0__0__20221129T1419Z] [testtable__0__0__20221129T1419Z] Could not commit segment. Retrying after hold
2022/11/29 16:11:00.997 INFO [HttpClient] [testtable__2__0__20221129T1419Z] Sending request: http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentConsumed?reason=forceCommitMessageReceived&streamPartitionMsgOffset=%7B%22shardId-000000000002%22%3A%2249635630746998378075227930204125252036295126557430644770%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__2__0__20221129T1419Z&rowCount=5136&memoryUsedBytes=1828440 to controller: SERVER_IP_ADDRESS.ec2.internal, version: Unknown
2022/11/29 16:11:00.997 INFO [ServerSegmentCompletionProtocolHandler] [testtable__2__0__20221129T1419Z] Controller response {"offset":-1,"isSplitCommitType":false,"streamPartitionMsgOffset":"{\"shardId-000000000002\":\"49635630746998378075227930204125252036295126557430644770\"}","buildTimeSec":-1,"status":"HOLD"} for http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentConsumed?reason=forceCommitMessageReceived&streamPartitionMsgOffset=%7B%22shardId-000000000002%22%3A%2249635630746998378075227930204125252036295126557430644770%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__2__0__20221129T1419Z&rowCount=5136&memoryUsedBytes=1828440
2022/11/29 16:11:01.474 INFO [HttpClient] [testtable__1__0__20221129T1419Z] Sending request: http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentConsumed?reason=forceCommitMessageReceived&streamPartitionMsgOffset=%7B%22shardId-000000000001%22%3A%2249635630746976077330029399572174273870489207002061864978%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__1__0__20221129T1419Z&memoryUsedBytes=1115986 to controller: SERVER_IP_ADDRESS.ec2.internal, version: Unknown
2022/11/29 16:11:01.474 INFO [ServerSegmentCompletionProtocolHandler] [testtable__1__0__20221129T1419Z] Controller response {"offset":-1,"isSplitCommitType":true,"controllerVipUrl":"http://localhost:9000","streamPartitionMsgOffset":"{\"shardId-000000000001\":\"49635630746976077330029399572174273870489207002061864978\"}","buildTimeSec":126,"status":"COMMIT"} for http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentConsumed?reason=forceCommitMessageReceived&streamPartitionMsgOffset=%7B%22shardId-000000000001%22%3A%2249635630746976077330029399572174273870489207002061864978%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__1__0__20221129T1419Z&memoryUsedBytes=1115986
2022/11/29 16:11:01.476 INFO [HttpClient] [testtable__1__0__20221129T1419Z] Sending request: http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentCommitStart?segmentSizeBytes=5072&buildTimeMillis=5&streamPartitionMsgOffset=%7B%22shardId-000000000001%22%3A%2249635630746976077330029399572174273870489207002061864978%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__1__0__20221129T1419Z&memoryUsedBytes=1115986 to controller: SERVER_IP_ADDRESS.ec2.internal, version: Unknown
2022/11/29 16:11:01.476 INFO [ServerSegmentCompletionProtocolHandler] [testtable__1__0__20221129T1419Z] Controller response {"offset":-1,"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE"} for http://SERVER_IP_ADDRESS.ec2.internal:9000/segmentCommitStart?segmentSizeBytes=5072&buildTimeMillis=5&streamPartitionMsgOffset=%7B%22shardId-000000000001%22%3A%2249635630746976077330029399572174273870489207002061864978%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__1__0__20221129T1419Z&memoryUsedBytes=1115986
2022/11/29 16:11:01.476 ERROR [LLRealtimeSegmentDataManager_testtable__1__0__20221129T1419Z] [testtable__1__0__20221129T1419Z] Could not send request http://localhost:9000/segmentUpload?segmentSizeBytes=5072&buildTimeMillis=5&streamPartitionMsgOffset=%7B%22shardId-000000000001%22%3A%2249635630746976077330029399572174273870489207002061864978%22%7D&instance=Server_SERVER_IP_ADDRESS.ec2.internal_8098&offset=-1&name=testtable__1__0__20221129T1419Z&memoryUsedBytes=1115986
org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:9000 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.common.utils.http.HttpClient.sendRequest(HttpClient.java:276) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegment(FileUploadDownloadClient.java:604) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.Server2ControllerSegmentUploader.uploadSegmentToController(Server2ControllerSegmentUploader.java:76) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.Server2ControllerSegmentUploader.uploadSegment(Server2ControllerSegmentUploader.java:61) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.SplitSegmentCommitter.uploadSegment(SplitSegmentCommitter.java:78) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.SplitSegmentCommitter.commit(SplitSegmentCommitter.java:59) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.commit(LLRealtimeSegmentDataManager.java:1025) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.commitSegment(LLRealtimeSegmentDataManager.java:995) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:711) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.apache.pinot.shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-58c1dc9b7d8e28ac7207fa7565b1ed76f7a0c2ad]
... 19 more
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.