apache / apache/beam

Add Schema support for Tensorflow

Open
#19,721 0 comments 0 reactions 0 assignees View on GitHub
improvement io java P3 tfrecord
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Similar to BEAM-5807, Tensorflow's defacto storage format is `TFRecord` files with `Example` proto payload and its own schema.proto. We already have `TFRecordIO` support. Need to implement:
* Conversion between Beam and TF schema
* Conversion between Beam Row and TF Example proto
* `TFRecordTableProvider`

[https://github.com/tensorflow/metadata/blob/master/tensorflow_metadata/proto/v0/schema.proto](https://github.com/tensorflow/metadata/blob/master/tensorflow_metadata/proto/v0/schema.proto)

[https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto)

 

Also it seems the metadata protos are not published as Java artifacts:

[https://github.com/tensorflow/metadata/issues/5](https://github.com/tensorflow/metadata/issues/5)

My WIP branch: https://github.com/spotify/beam/tree/neville/tf

Imported from Jira [BEAM-7921](https://issues.apache.org/jira/browse/BEAM-7921). Original Jira may contain additional context.
Reported by: sinisa_lyh.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.