a8m / a8m/kinesis-producer

Accept an interface instead of []byte in producer's Put method

Đang mở
#11 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Go
Star
150
Fork
47
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

Put expects records to be written to Kinesis as []byte. This works fine for simple cases but often a more complex layer wraps kinesis-producer and needs more control/flexibility over how different cases (such as failures) are handled. For example, a program might want to handle failures in a special way but right now the FailureResult struct only contains the partition key and the raw data as bytes. There is no other identifying information that might help identify the records.

Accepting an interface instead of raw bytes would make the library a lot more flexible. For example,

```go
type Record struct {
id string
data []byte
failure_count int
}

func (r Record) Bytes() []byte {
return r.data
}

func process(r Record) {
producer.Put(r, "partition-key")
}

func processErrors() {
for f := producer.NotifyFailures() {
r := f.Item.(Record)
if (r.failure_count > 3) {
// log failure
return
}
r.failure_count++
process(r)
}
}
```

Or a record could marshal itself as bytes that kinesis-producer could use internally by calling the Read() method and then wouldn't have to unmarshal the bytes on every failure in order to be able to inspect the record (for specialized logging, retries, failure handling etc).

kinesis-producer API would add an interface definition

```go
type Record interface {
Read() []byte
}

producer.Put(r Record, k string)

type FailureRecord struct {
Record Record
ParitionKey string
}
```

Or partition key could be rolled into the record as well like:

```go
type Record interface {
Read() []byte
PartitionKey() string
}

producer.Put(r Record)
```

In this case, failure channel would just return failed Records and client side implementation could look like:

```go
type Record struct {
ID string
UserID string
Timestamp time.Time
// any other number of fields
}

func (r Record) Read() []byte {
// marshal r and return []bytes
}

func (r Record) PartitionKey() string {
return r.ID
}
```

The library could also ship with a simple implementation to cover simple cases. Usage would look like:

```go
pr.Put(producer.Record{myData, myPartitionKey})
```

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.