linkedin / linkedin/goavro

Getting avro message metadata

Open
#145 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
1.1k
Forks
232
PR merge metrics
No merged PRs in 30d

Description

In my system I need to get `Schema` from Avro message, and I notice we could get `Schema` by calling `(*OCFReader).Metadata()`. But when generating new `OCFReader` it has to generate new `codec`, and I see this is quite expensive to do. Ref: https://github.com/linkedin/goavro/blob/master/ocf.go#L114-L171

So, I propose to create function that will return Avro metadata when called, without creating new `OCFReader`. This is what I used now:
```
func readAvroHeader(ctx context.Context, ior io.Reader) (map[string][]byte, error) {
span, ctx := tracer.StartSpanFromContext(ctx)
defer span.Finish()

//
// magic bytes
//
magic := make([]byte, 4)
_, err := io.ReadFull(ior, magic)
if err != nil {
return nil, fmt.Errorf("cannot read OCF header magic bytes: %s", err)
}

if !bytes.Equal(magic, ocfMagicBytes) {
return nil, fmt.Errorf("cannot read OCF header with invalid magic bytes: %#q", magic)
}

//
// metadata
//
metadata, err := metadataBinaryReader(ctx, ior)
if err != nil {
return nil, fmt.Errorf("cannot read OCF header metadata: %s", err)
}
return metadata, nil
}
```

What do you guys think?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review ocf.go around the OCFReader constructor and Metadata method, then inspect metadataBinaryReader and the proposed header-reading flow. Define an accessible metadata path that avoids constructing a new codec or OCFReader, and verify that it reads the OCF magic bytes and metadata with the existing error behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.