tensorflow / tensorflow/datasets

How to load a key with values of different tyeps in tfrecord?

Open
#4,736 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

What I need help with / What I was wondering
I have some third-party generated tfrecord files. I just found there is a specific key that has different value types in these tfrecord files, shown as follows.

key: "similarity"
    value {
      float_list {
        value: 0.3015786111354828
      }
    }
key: "similarity"
    value {
        bytes_list {
          value: ""
        }
    }

When I try to decode this key-value pair in tfrecord, I encounter a problem. I cannot find the suitable type for this key similarity. When I use tf.string or tfds.features.Text() in tfds.features.FeaturesDict for decoding, it returns the error

Data types don't match. Data type: float but expected type: string

When I use tf.float64 in tfds.features.FeaturesDict for decoding, it returns the error

Data types don't match. Data type: string but expected type: float

I wonder if there is anything in tfds.features or tf.train.Example that allows me to decode both float and string?

Or if there is something like tfds.decode.SkipDecoding() that allows me read this key similarity and decide how to decode it afterwards? I am aware that tfds.builder().as_dataset() has that option, but I cannot find one in tf.data.TFRecordDataset. I have tried to simply remove the entry correspondind to the key similarity, but the data read from the tfrecord dataset simply drop the entry similarity.

Thanks a lot!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tfds.features.FeaturesDict, tf.train.Example, and tf.data.TFRecordDataset to trace how feature types are validated and whether decoding can be deferred. Done means establishing the supported behavior for mixed float and bytes values and documenting or exposing an appropriate handling path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.