a8m / a8m/kinesis-producer

Accept an interface instead of []byte in producer's Put method

Open
#11 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
150
Forks
47
PR merge metrics
No merged PRs in 30d

Description

Put expects records to be written to Kinesis as []byte. This works fine for simple cases but often a more complex layer wraps kinesis-producer and needs more control/flexibility over how different cases (such as failures) are handled. For example, a program might want to handle failures in a special way but right now the FailureResult struct only contains the partition key and the raw data as bytes. There is no other identifying information that might help identify the records.

Accepting an interface instead of raw bytes would make the library a lot more flexible. For example,

```go
type Record struct {
id string
data []byte
failure_count int
}

func (r Record) Bytes() []byte {
return r.data
}

func process(r Record) {
producer.Put(r, "partition-key")
}

func processErrors() {
for f := producer.NotifyFailures() {
r := f.Item.(Record)
if (r.failure_count > 3) {
// log failure
return
}
r.failure_count++
process(r)
}
}
```

Or a record could marshal itself as bytes that kinesis-producer could use internally by calling the Read() method and then wouldn't have to unmarshal the bytes on every failure in order to be able to inspect the record (for specialized logging, retries, failure handling etc).

kinesis-producer API would add an interface definition

```go
type Record interface {
Read() []byte
}

producer.Put(r Record, k string)

type FailureRecord struct {
Record Record
ParitionKey string
}
```

Or partition key could be rolled into the record as well like:

```go
type Record interface {
Read() []byte
PartitionKey() string
}

producer.Put(r Record)
```

In this case, failure channel would just return failed Records and client side implementation could look like:

```go
type Record struct {
ID string
UserID string
Timestamp time.Time
// any other number of fields
}

func (r Record) Read() []byte {
// marshal r and return []bytes
}

func (r Record) PartitionKey() string {
return r.ID
}
```

The library could also ship with a simple implementation to cover simple cases. Usage would look like:

```go
pr.Put(producer.Record{myData, myPartitionKey})
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.