heetch / heetch/avro

support stream encoding/decoding

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
95
Forks
18
PR merge metrics
No merged PRs in 30d

Description

We'd like to support OCF files, and potentially other kinds of streamed Avro record files.

Possible API:

```
// NewEncoder returns a new Encoder instance that encodes
// a stream of messages into r using the given schema.
// All messages encoded must have the same Avro schema.
//
// By default, it encodes the Avro object container file (OCF) format,
// writing the schema header when the first value is encoded.
func NewStreamEncoder(w io.Writer, schema string) *StreamEncoder

// Encode writes a message to the encoder.
// All messages must have the same Avro schema.
func (enc *StreamEncoder) Encode(x interface{}) error

// OmitHeader specifies that no file header should be written,
// just the encoded values themselves without the schema.
func (enc *StreamEncoder) OmitHeader()

type StreamDecoder struct {
readerSchema schema.AvroType
progs map[reflect.Type]*program
}

// NewStreamDecoder returns a new StreamDecoder instance that
// decodes a stream of messages from r.
// By default it decodes the Avro object container
// file (OCF) format, setting the schema by reading it
// from the start of the file.
// See https://avro.apache.org/docs/1.8.2/spec.html#Object+Container+Files
func NewStreamDecoder(r io.Reader) *StreamDecoder

// SetSchema sets the schema used to decode the
// message stream. If this is called before any messages
// are read, the decoder will not read the OCF
// header.
func (dec *StreamDecoder) SetSchema(schema string) error

// Buffered returns a reader of the data remaining in the Decoder's
// buffer. The reader is valid until the next call to Decode.
func (dec *StreamDecoder) Buffered() io.Reader

// Decode decodes the next value into x, which should
// be a pointer to an Avro-compatible type (see
// Unmarshal for further details).
func (dec *StreamDecoder) Decode(x interface{}) error
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.