possiblyResetDataSourceMetadata() slows down the kinesis index task
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
In all versions
### Description
Unlike the kafka task, the kinesis task implements `possiblyResetDataSourceMetadata()` and checks the availability of the current offsets by comparing them with the earliest offsets available in the stream. This can take a long time if Kinesis is busy as shown in the below flame graph.

`AmazonKinesis.getShardIterator()` throws an `InvalidArgumentException` if the `startingSequenceNumber` is not available in the stream. Similar to the kafka task (it calls `possiblyResetOffsetsOrWait()` when `recordSupplier.poll()` throws an `OffsetOutOfRangeException`), we can call `possiblyResetDataSourceMetadata()` when `InvalidArgumentException` is thrown. However, since the `InvalidArgumentException` can be thrown for various reasons, we need to make sure that was thrown because of the invalid `startingSequenceNumber`.
Contributor guide
Research direction
Inspect the Kinesis task's possiblyResetDataSourceMetadata() path and the recordSupplier.poll() error handling, then compare it with the Kafka task's possiblyResetOffsetsOrWait() behavior. Determine how InvalidArgumentException identifies an unavailable starting sequence number, and consider the change complete when metadata reset is deferred until that condition occurs rather than on every check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100