apache / apache/druid

Kafka Indexing task pausing forever if no data received in intermediateHandoffPeriod

Open
#6,968 3 comments 0 reactions 0 assignees View on GitHub
Area - Streaming Ingestion Bug stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

druid: 0.13.0
Kafka: 1.1.1

I've been running a number of tests locally with Kafka indexing and I believe something similar to what this was meant to fix https://github.com/apache/incubator-druid/commit/638f50cb52c248f4408975d5fc7762cc9ce82d8e is still occurring.

I've set the intermediateHandoffPeriod to a low value PT5M while testing to see what handoffs and shards will look like.

If I send data in every < 5 minutes then the task continues indexing. However if there is a gap where no data lands during an intermediate hand off period I see this logged out in the task:

```
2019-01-30T14:18:16,501 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_kafka_testTopic_881471c82f88076_caeelnij]: CheckPointDataSourceMetadataAction{supervisorId='testTopic', baseSequenceName='index_kafka_testTopic_881471c82f88076', taskGroupId='0', previousCheckPoint=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic='testTopic', partitionOffsetMap={0=3313}}}, currentCheckPoint=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic='testTopic', partitionOffsetMap={0=3313}}}}
2019-01-30T14:18:16,501 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_testTopic_881471c82f88076_caeelnij] to overlord: [CheckPointDataSourceMetadataAction{supervisorId='testTopic', baseSequenceName='index_kafka_testTopic_881471c82f88076', taskGroupId='0', previousCheckPoint=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic='testTopic', partitionOffsetMap={0=3313}}}, currentCheckPoint=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic='testTopic', partitionOffsetMap={0=3313}}}}].
2019-01-30T14:18:16,507 INFO [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Pausing ingestion until resumed
2019-01-30T14:18:16,512 INFO [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Pausing ingestion until resumed
2019-01-30T14:18:16,520 WARN [qtp873134840-79] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Ignoring duplicate request, end offsets already set for sequences [[SequenceMetadata{sequenceName='index_kafka_testTopic_881471c82f88076_2', sequenceId=2, startOffsets={0=3313}, endOffsets={0=9223372036854775807}, assignments=[0], sentinel=false, checkpointed=false}]]
```
and any further data sent to the stream is not indexed. If the supervisor task is suspended and resumed, then the task starts reading and indexing the kafka stream from the last checkpoint, and no data loss is seen.

Contributor guide

Open the contributing guide

Research direction

Reproduce with Druid 0.13.0 and Kafka 1.1.1 using an intermediateHandoffPeriod of PT5M, leaving one handoff period without new data. Start with IncrementalPublishingKafkaIndexTaskRunner and the logged checkpoint and duplicate-request messages. Done means data sent after the empty period is indexed without suspending and resuming the supervisor, with no data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.