linkedin / linkedin/spark-tfrecord
Dataframe can not convert to tfrecord
Open
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 300
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
val df1: DataFrame = spark.createDataFrame(rdd,subSchema)
val df2 = df1.withColumn("entity",struct("age","salary")).
groupBy("employee_name")
df1 can convert to tfrecord,but df2 cannot .
why is that?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the conversion with the two DataFrame examples in the issue, comparing df1 with the grouped df2 containing the entity struct. Trace the DataFrame-to-TFRecord entry point and determine whether the grouped or nested schema is supported; document the limitation or define the expected corrected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, tensorflow
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100