linkedin / linkedin/spark-tfrecord
ArrayType(ArrayType(DoubleType,true),true) Not Supported When Writing Dataframe to TFRecords
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 300
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I try to write my dataframe to tfrecords but encounter the error, log is as below
Caused by: java.lang.RuntimeException: Cannot convert field to unsupported data type ArrayType(ArrayType(DoubleType,true),true)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$.org$tensorflow$spark$datasources$tfrecords$serde$DefaultTfRecordRowEncoder$$encodeFeature(DefaultTfRecordRowEncoder.scala:144)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$$anonfun$encodeExample$1.apply(DefaultTfRecordRowEncoder.scala:64)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$$anonfun$encodeExample$1.apply(DefaultTfRecordRowEncoder.scala:61)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.tensorflow.spark.datasources.tfrecords.serde.DefaultTfRecordRowEncoder$.encodeExample(DefaultTfRecordRowEncoder.scala:61)
at org.tensorflow.spark.datasources.tfrecords.DefaultSource$$anonfun$2.apply(DefaultSource.scala:59)
at org.tensorflow.spark.datasources.tfrecords.DefaultSource$$anonfun$2.apply(DefaultSource.scala:56)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:129)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
... 10 more
I presume such feature is already supported though it's not specifically addressed in README
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in DefaultTfRecordRowEncoder.scala, especially encodeFeature at line 144 and encodeExample at lines 61-64, where the unsupported nested ArrayType is rejected. Review the README's supported types and determine the expected TFRecord representation for ArrayType(ArrayType(DoubleType,true),true). Done means writing the reported dataframe no longer fails for this type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, tensorflow
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100