apache / apache/gravitino

[FEATURE] Is there a better way to manage schema of tfrecord

Open
#2,885 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Describe the feature

We have many tfrecords files on hdfs, these files were created by spark and treated as hadoop files. Currently, we face some issues:
1. The files are scattered on hdfs, since there is no centralized management, many outdated files remain and occupy storage space.
2. We have many analysis needs on these tfrecords files. Currently, we can query these files via spark-tfrecord-connector provided by tensorflow [ecosystem](https://github.com/tensorflow/ecosystem) or [spark-tfrecord](https://github.com/linkedin/spark-tfrecord). the code like below
```
create temporary view tf_view using tfrecord options(path "file:/xxx/tfrecord");

select * from tf_view;
```
But there are some issues with these connectors. One problem is that the tfrecord schema isn't persistent. Every query will take long time to infer schema, because spark need scan a large size file or all files to desearialize `tf.Example`. `tf.Example` is a message defined by Protocol Buffers (Protobuf) [tf.example proto](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/core/example/example.proto#L88) that represents a mapping of strings to byte values. A TFRecord file stores serialized instances of tf.Example.
In general, we only focus on the key-value pair in the map of `tf.Example`. Each key-value pair represents a feature name and its corresponding value.

It would be nice if we can manage these files including schema and provide a compatible spark connector for query.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Spark TFRecord connector examples and the linked TensorFlow Example protobuf definition to understand the current schema-inference path. The issue does not identify repository files, tests, or a settled implementation; done would require an agreed design for centralized TFRecord and schema management plus a compatible Spark query connector.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, spark, tensorflow
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.