linkedin / linkedin/spark-tfrecord
spark 读取 tfrecord文件,解析tf.train.Feature(bytes_list=tf.train.BytesList(np.array(feature, dtype=np.float32).tobytes())) 文件字节码异常
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 300
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
1. 如下图所示:用spark直接去读取tfrecord数据,string类型的字段可以直接读取,但是adc_traj等numpy.array的数据会是乱码的形式。
2. 原因是spark加载tfrecord数据的时候,会默认把bytes类型的数据转成str,目前不清楚字符编码。而adc_traj存储的时候是用tf.train.Feature(bytes_list=tf.train.BytesList(np.array(feature, dtype=np.float32).tobytes()))方式存储的。
3. 用 np.frombuffer(str_y., dtype=np.float32)去解析数据
数据还是会报错:
ValueError: buffer size must be a multiple of element size
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is identified. Reproduce the Spark TFRecord read using the bytes_list value created from a float32 NumPy array, then trace where bytes become a string and compare the original byte length with np.frombuffer's input. Done means binary feature data can be decoded without the buffer-size error, with a regression test covering this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, tensorflow
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100