Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

What happens if first record does not have headers but further records in the same batch do?

Open
#227 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
39
Avg merge
2d 10h
Merged PRs (30d)
5

Description

Output schema (avro/parquet) is based on the first record. This works fine for key/value, etc. but headers may differ between records. Different record header elements types can cause issues but this can be solved by casting header types by using StringConverter or similar for headers.
Though, when first record does not have headers the schema type is null. We should figure out what happens with next records in the same batch if they do have header (are the headers missing? does conversion fail?) and provide an approach on how to process infer the schema for headers properly.

See https://github.com/Aiven-Open/gcs-connector-for-apache-kafka/issues/347#issuecomment-1964537795

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.