apache / apache/beam

Avro-Protobuf support

Open
#18,965 0 comments 0 reactions 0 assignees View on GitHub
avro io java P3 wish
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I came a cross avro-protobuf support which allows to write Protobuf messages to avro file and then read them back as a) GenericRecords or b) as Protobuf typed message

Quite nice feature I am looking for. I do not know much about what is happening behind the scenes, if there are not some hidden problems and limitations e.g. on schema compatibility when evolving proto messages etc.

So what I want to ask is:
- Was this feature ever considered, to enable existing AvroIO code to use also ProtobufDatumReader and ProtobufDatumWriter? Any wisdom what I should be aware of?
- Would it make sense to add it to AvroIO ?

 

P.S.: I wanted to add it on the side for my needs and testing it but I did not find an easy way of doing it. Writer infrastructure is easy to extend by implementing custom FileIO.Sink but read side seems to be hard nut to crack without copy-pasting much code from AvroIO and AvroSource.

Imported from Jira [BEAM-4758](https://issues.apache.org/jira/browse/BEAM-4758). Original Jira may contain additional context.
Reported by: JozoVilcek.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.