[FEATURE] Is there a better way to manage schema of tfrecord
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### Describe the feature
We have many tfrecords files on hdfs, these files were created by spark and treated as hadoop files. Currently, we face some issues:
1. The files are scattered on hdfs, since there is no centralized management, many outdated files remain and occupy storage space.
2. We have many analysis needs on these tfrecords files. Currently, we can query these files via spark-tfrecord-connector provided by tensorflow [ecosystem](https://github.com/tensorflow/ecosystem) or [spark-tfrecord](https://github.com/linkedin/spark-tfrecord). the code like below
```
create temporary view tf_view using tfrecord options(path "file:/xxx/tfrecord");
select * from tf_view;
```
But there are some issues with these connectors. One problem is that the tfrecord schema isn't persistent. Every query will take long time to infer schema, because spark need scan a large size file or all files to desearialize `tf.Example`. `tf.Example` is a message defined by Protocol Buffers (Protobuf) [tf.example proto](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/core/example/example.proto#L88) that represents a mapping of strings to byte values. A TFRecord file stores serialized instances of tf.Example.
In general, we only focus on the key-value pair in the map of `tf.Example`. Each key-value pair represents a feature name and its corresponding value.
It would be nice if we can manage these files including schema and provide a compatible spark connector for query.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing the Spark TFRecord connector examples and the linked TensorFlow Example protobuf definition to understand the current schema-inference path. The issue does not identify repository files, tests, or a settled implementation; done would require an agreed design for centralized TFRecord and schema management plus a compatible Spark query connector.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, spark, tensorflow
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100