apache / apache/druid

possiblyResetDataSourceMetadata() slows down the kinesis index task

Open
#9,955 0 comments 0 reactions 0 assignees View on GitHub
Area - Streaming Ingestion Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

In all versions

### Description

Unlike the kafka task, the kinesis task implements `possiblyResetDataSourceMetadata()` and checks the availability of the current offsets by comparing them with the earliest offsets available in the stream. This can take a long time if Kinesis is busy as shown in the below flame graph.

![Screen Shot 2020-05-29 at 2 29 47 PM](https://user-images.githubusercontent.com/2322288/83322343-ac765e80-a20b-11ea-94fc-5e420501c323.png)

`AmazonKinesis.getShardIterator()` throws an `InvalidArgumentException` if the `startingSequenceNumber` is not available in the stream. Similar to the kafka task (it calls `possiblyResetOffsetsOrWait()` when `recordSupplier.poll()` throws an `OffsetOutOfRangeException`), we can call `possiblyResetDataSourceMetadata()` when `InvalidArgumentException` is thrown. However, since the `InvalidArgumentException` can be thrown for various reasons, we need to make sure that was thrown because of the invalid `startingSequenceNumber`.

Contributor guide

Open the contributing guide

Research direction

Inspect the Kinesis task's possiblyResetDataSourceMetadata() path and the recordSupplier.poll() error handling, then compare it with the Kafka task's possiblyResetOffsetsOrWait() behavior. Determine how InvalidArgumentException identifies an unavailable starting sequence number, and consider the change complete when metadata reset is deferred until that condition occurs rather than on every check.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.