tensorflow / tensorflow/datasets
How to load a key with values of different tyeps in tfrecord?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
What I need help with / What I was wondering
I have some third-party generated tfrecord files. I just found there is a specific key that has different value types in these tfrecord files, shown as follows.
key: "similarity"
value {
float_list {
value: 0.3015786111354828
}
}
key: "similarity"
value {
bytes_list {
value: ""
}
}
When I try to decode this key-value pair in tfrecord, I encounter a problem. I cannot find the suitable type for this key similarity. When I use tf.string or tfds.features.Text() in tfds.features.FeaturesDict for decoding, it returns the error
Data types don't match. Data type: float but expected type: string
When I use tf.float64 in tfds.features.FeaturesDict for decoding, it returns the error
Data types don't match. Data type: string but expected type: float
I wonder if there is anything in tfds.features or tf.train.Example that allows me to decode both float and string?
Or if there is something like tfds.decode.SkipDecoding() that allows me read this key similarity and decide how to decode it afterwards? I am aware that tfds.builder().as_dataset() has that option, but I cannot find one in tf.data.TFRecordDataset. I have tried to simply remove the entry correspondind to the key similarity, but the data read from the tfrecord dataset simply drop the entry similarity.
Thanks a lot!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tfds.features.FeaturesDict, tf.train.Example, and tf.data.TFRecordDataset to trace how feature types are validated and whether decoding can be deferred. Done means establishing the supported behavior for mixed float and bytes values and documenting or exposing an appropriate handling path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100