Delete records via the Producer API

Start with Redpanda’s Kafka-compatible Producer API and the ProducerRecord header proposal, then compare the requested behavior with Kafka’s admin client record-deletion API. No implementation file or test is named. Done means a programmer can request deletion before a specified topic, partition, and offset through the client API, with behavior defined for the affected records.

Who is this for and what problem do they have today?

A long standing problem I’ve had with the Kafka API is that topic/partition purging appears to be the exclusivity of an admin. For example, in Apache Kafka, we can use the admin client API to delete records prior to a partition/offset tuple. I also note that Redpanda has a feature request to support this: https://github.com/vectorizedio/redpanda/issues/2648

My particular issue with topic/partition purging is that it doesn’t appear to be able to be a programmer’s concern. In my ideal world as a programmer, I would like to be able to purge a topic through the regular Kafka API as I’m probably in a better position than admin to know when data can be purged. Discussions around Kafka record deletion often appear to be focused on admin use-cases wanting to free up data. That’s fine, but I want to make sure that no more messages than are necessary for my application need exist given a topic/partition tuple. Any new node coming online that needs to catch up on consuming can then avoid trawling through events that will become discarded given subsequent events.

In summary, there is no way for Redpanda to know when it is best to purge records from a topic. Here's a use case that cannot be satisfied.

1. I have 100 NameChanged events in my topic, which is able to contain other types of event also.
2. I'm only ever interested in the last 2 name changes.
3. I want to retain different numbers of other event types within the topic.

One particular production scenario where I was using another commit log implementation was when an IoT sensor became faulty. This sensor then emitted thousands of events in relation to want to join the network. The topic quickly became saturated with these events, and while storage remained fine, the consumers of this data spent many minutes trawling through the data. While this situation may not be entirely avoidable, having the ability for some background task to cleanup would minimise the pain point.

What are the success criteria?

Allowing the programmer to express that all records prior to a given offset and partition can be removed using the client API.

Why is solving this problem impactful?

The existing policies of topic retention and compaction are limited and also admin oriented. The programmer is in the best position to convey the lifetime of records.

Not having this issue impacts production systems given undesirable data in a topic. Consumers of these topics are heavily impacted.

Additional notes

I went searching for a KIP in relation to this, but couldn't find one. Perhaps this should become a KIP request. However, a simple non-breaking API change would be to be able to allow the programmer to add special headers of Redpanda to a ProducerRecord. A special header could be used to convey the offset for Redpanda to delete records up to with the associated topic and partition. A programmer would typically convey a value in relation to the snapshot of state for this record.

JIRA Link: CORE-862