Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka
What happens if first record does not have headers but further records in the same batch do?
- Dominant language
- Java
- Stars
- 58
- Forks
- 39
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 5
Description
Output schema (avro/parquet) is based on the first record. This works fine for key/value, etc. but headers may differ between records. Different record header elements types can cause issues but this can be solved by casting header types by using StringConverter or similar for headers.
Though, when first record does not have headers the schema type is null. We should figure out what happens with next records in the same batch if they do have header (are the headers missing? does conversion fail?) and provide an approach on how to process infer the schema for headers properly.
See https://github.com/Aiven-Open/gcs-connector-for-apache-kafka/issues/347#issuecomment-1964537795
Contributor guide
Assessment
This issue has not been assessed yet.