apache / apache/fluss

Support Kafka Protocol 4: Add utils to convert Kafka MemoryRecords

Open
#607 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Parent issue: https://github.com/alibaba/fluss/issues/486

As the Fluss protocols is different with Kafka protocol, so for the purpose of compatibility, we need a util to convert Kafka MemoryRecords to LogRecordBatch, and convert LogRecordBatch to Kafka MemoryRecords.

The util should be like:
```java
package com.alibaba.fluss.kafka;

import com.alibaba.fluss.record.LogRecordBatch;
import org.apache.kafka.common.record.MemoryRecords;

public class RecordsUtil {

public static LogRecordBatch convert(MemoryRecords records) {
return null;
}

public static MemoryRecords convert(LogRecordBatch batch) {
return null;
}

}
```

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with parent issue #486 and the proposed com.alibaba.fluss.kafka.RecordsUtil API, then inspect LogRecordBatch and Kafka's MemoryRecords definitions to determine the required mappings. Done means both conversion directions support the compatibility use case and are covered by verification for representative records.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.