apache / apache/beam

[Bug]: KafkaIO missing initial messages when using automatic topic creation

Open
#27,246 1 comment 0 reactions 0 assignees View on GitHub
bug java kafka P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

I noticed missing sequences when using `GenerateSequence + KafkaIO` to produce records to Kafka, when relying on the automatic topic creation feature (auto.create.topics.enable).

What I've used to reproduce:

```
Pipeline p = Pipeline.create(pipelineOptions);

PCollection numbers =
p.apply(GenerateSequence.from(0).withRate(100, Duration.millis(100)));
numbers
.apply(
MapElements.into(
TypeDescriptors.kvs(TypeDescriptors.longs(), TypeDescriptors.strings()))
.via(x -> KV.of(x, String.valueOf(x))))
.apply(
KafkaIO.write()
.withTopic(TOPIC_NAME)
.withBootstrapServers(BOOTSTRAP_SERVERS)
.withKeySerializer(LongSerializer.class)
.withValueSerializer(StringSerializer.class));

p.run();
```

And consuming using KafkaIO.read() + writing to a file:

```
Pipeline p = Pipeline.create(pipelineOptions);

List partitions = new ArrayList<>();
for (int i = 0; i < 20; i++) {
partitions.add(new TopicPartition(TOPIC_NAME, i));
}

PCollection>> ticks = p.apply(
KafkaIO.read()
.withBootstrapServers(BOOTSTRAP_SERVERS)
.withKeyDeserializer(LongDeserializer.class)
.withValueDeserializer(StringDeserializer.class)
.withTopicPartitions(partitions))
.apply(MapElements.into(TypeDescriptors.strings()).via(record -> record.getKV().getValue()));

ticks.apply("Window", Window.into(FixedWindows.of(Duration.standardMinutes(1))))
.apply(
TextIO.write().to("gs://test-bucket/" + TOPIC_NAME + "/keys-")
.withNumShards(0)
.withWindowedWrites());

p.run();
```

Then I copy the files from gcs (`gsutil -m cp gs://test-bucket/{TOPIC}/* /tmp/test`) and created a simple class to check for continuity:

```
public class TestContiguous {

public static void main(String[] args) {
findMissingNumbersInFiles(new File("/tmp/example-stream/"));
}

public static List findMissingNumbersInFiles(File folder) {
File[] files = folder.listFiles();

TreeSet numbers = new TreeSet<>();
for (File file : files) {
numbers.addAll(readNumbersFromFile(file));
}

List missingNumbers = new ArrayList<>();
int minNumber = numbers.first();
int maxNumber = numbers.last();

for (int i = minNumber; i <= maxNumber; i++) {
if (!numbers.contains(i)) {
missingNumbers.add(i);
}
}

System.out.println(
"Min: " + minNumber + ", Max: " + maxNumber + ". Missing: " + missingNumbers);
return missingNumbers;
}

public static List readNumbersFromFile(File file) {
List numbers = new ArrayList<>();
try (Scanner scanner = new Scanner(file)) {
while (scanner.hasNextLine()) {
String line = scanner.nextLine().trim();
if (!line.isEmpty()) {
try {
int number = Integer.parseInt(line.split(",")[0]);
numbers.add(number);
} catch (NumberFormatException e) {
e.printStackTrace();
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return numbers;
}

}
```

It is expected that there will be some records missing at the tail end due to parallelism, but I've always encountered about ~1k of sequences missing at the beginning of the pipeline:

```
Min: 1268, Max: 780327. Missing: [1269, 1270, 1271, 1272, 1273, 1276, 1277, 1280, 1282, 1283, 1284, 1293, 780319, 780322, 780323, 780324, 780326]
```

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.