apache / apache/pinot

Kinesis ApiCallTimeout during the ingestion

Open
#7,238 4 comments 0 reactions 0 assignees View on GitHub
kinesis
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

i'm using pinot 0.8.0 master branch.

Currently my producer is sending the records for 10 seconds and in the mean time the producer is idle and its throwing out the below error. same error is being thrown continuously for every 30sec, event after producer stopped.

```
2021/08/01 18:01:30.705 INFO [LLRealtimeSegmentDataManager_realtime_events__9__0__20210801T1750Z] [realtime_events__9__0__20210801T1750Z] Consumed 0 events from (rate:0.0/s), currentOffset={"shardId-000000000009":"49619229921359620555077470674058986369313634779052114066"}, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0
2021/08/01 18:01:30.705 WARN [KinesisConsumer] [pool-22-thread-1] Task aborted due to exception.
shaded.software.amazon.awssdk.core.exception.AbortedException: Thread was interrupted
at shaded.software.amazon.awssdk.core.exception.AbortedException$BuilderImpl.build(AbortedException.java:84) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.exception.AbortedException.create(AbortedException.java:38) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:143) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at shaded.software.amazon.awssdk.services.kinesis.DefaultKinesisClient.getRecords(DefaultKinesisClient.java:1050) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at org.apache.pinot.plugin.stream.kinesis.KinesisConsumer.getResult(KinesisConsumer.java:131) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at org.apache.pinot.plugin.stream.kinesis.KinesisConsumer.lambda$fetchMessages$0(KinesisConsumer.java:86) ~[pinot-kinesis-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-78b08e4af41adc2e5e6d7387960df0524e2a7e90]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```

could some one please check. thanks

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.pinot.plugin.stream.kinesis.KinesisConsumer, especially fetchMessages at line 86 and getResult at line 131, then trace the AWS Kinesis getRecords call shown in the stack trace. Reproduce the idle-producer case and determine whether repeated interrupted calls are expected; done should mean the idle ingestion behavior and its logging are corrected or clearly covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.