linkedin / linkedin/spark-tfrecord
java.lang.ClassNotFoundException: Failed to find data source: tfrecord. Please find packages at http://spark.apache.org/third-party-projects.html
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 300
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Hello authors,there exits some confuse when i try to run :
df.write.format("tfrecord").save("hdfs://***/a")
java.lang.ClassNotFoundException: Failed to find data source: tfrecord. Please find packages at http://spark.apache.org/third-party-projects.html
And it's confuse that when i remove spark-mllib dependency the error mistakes . it's is any conflict between mllib and spark-tfrecord? I will appreciate that if you can handle my problem.Thanks again!
my dependency settings:
<java.version>1.8</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
UTF-8
<scala.version>2.11.12</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<spark.version>2.2.0</spark.version>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.binary.version}</artifactId>
<version>2.4.3</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.binary.version}</artifactId>
<version>2.4.3</version>
</dependency>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the df.write.format("tfrecord") call and inspect the Maven dependency settings shown in the report, including the Spark and Scala versions and duplicate spark-mllib entries. Check the resolved runtime classpath for the TFRecord data source; done means explaining the dependency conflict or missing package and confirming a working configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, tensorflow
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100