linkedin / linkedin/goavro

Proposal: Implement code generation with schema evolution support

Open
#109 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.1k
Forks
232
PR merge metrics
No merged PRs in 30d

Description

This is a proposal and shout out for help and collaboration.

I believe it will be a good benefit for Go community having a "standard" library for handling Avro data (I think something developed under LinkedIn name can be a good candidate). But I really believe also, that without proper code generation, Avro has very little benefits for Go community, because of already very well established Protobuf and Thrift tooling out there.

I'm not entirely sure if code generation with support for schema evolution is even possible in Avro, but if I'm not mistaken, Java generated code is supporting that, as long as you provide writer's schema (which is easy if you have schema registry).

So this proposal is a request for collaboration in order to develop a proper tool for generating Go code from Avro schemas, that would support schema evolution and would be idiomatic.

I'm willing to work on that, but don't have enough knowledge of internals and lower level things related to Avro.

Maybe just porting and adapting Java generated code can be a solution.

I'm asking for a little guidance here, and maybe this could fit into the scope of this library, so that we don't deliver things that won't be maintained afterward. I thing handling specific and generic avro records within one library could be a nice option.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reviewing the goavro library's existing encoding and decoding scope, then clarify the requirements for Go code generation and Avro schema evolution with maintainers; done would be an agreed, maintainable design and implemented generator.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.