Support KIP-280: Enhanced log compaction

Who is this for and what problem do they have today?

There is an accepted KIP to allow customisation of log compaction. In particular, it allows using either message header timestamp or a custom message header field instead of offset to determine which version to keep when compacting.

The implementation of the KIP in apache/kafka has been stalled since April 2020, so I'm not sure what this means for Redpanda implemeting the KIP. It would be great if it could be implemented despite it not yet being in Apache Kafka. If that's too risky, would implementing it behind a configuration flag that loudly states that the feature is experimental be possible?

The particular use case I would have is for a change data capture setting, eg using Debezium to stream changes from Postgres. The current way this works requires that publishers only publish in monotonically increasing order of the Postgres log sequence number (LSN). I would like to be able to indiscriminately publish potentially older versions of records from multiple replicas, and not have to worry about the log compaction resulting in the wrong version being kept. (Consumers would have custom logic to handle out-of-order versions in pre-compaction.)

What are the success criteria?

Enhanced log compaction is implemented as described in KIP-280.

Why is solving this problem impactful?

This is an accepted KIP and its implementation would have a lot of utility. I prefer to use Redpanda over Kafka for operational simplicity, so I'm most interested in it existing here.

Additional notes

References:

• KIP-280: Enhanced log compaction text
• Jira issue tracking implementation in Kafka
• Open pull request to apache/kafka implementing the feature

JIRA Link: CORE-963

Start by reading KIP-280 and the linked Kafka Jira issue and pull request to understand the proposed behavior and implementation status. Then assess how that design maps to Redpanda's C++ streaming implementation; done means enhanced log compaction works as described by KIP-280, including the stated CDC use case.