linkedin / linkedin/spark-tfrecord
Consume TensorFlowInferSchema from Java Spark 2.4.0
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 300
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
This is not an issue and is more of a request for guidance. I'd like to just use the TensorFlowInferSchema functionality in my Java spark job. Below is the sample snippet
JavaRDD<Example> exampleRdd =
jsc.parallelize(
Arrays.asList(
Example.newBuilder()
.setFeatures(
Features.newBuilder()
.putFeature(
"Test",
Feature.newBuilder()
.setInt64List(Int64List.newBuilder().addValue(10).build())
.build())
.build())
.build()));
StructType schema = TensorFlowInferSchema.apply(rdd.rdd(), ???);
However, the apply method expects a second argument - implicit evidence$1 : scala.reflect.runtime.universe.TypeTag[T].
Below is the compiled Java class
def apply[T](rdd : org.apache.spark.rdd.RDD[T])(implicit evidence$1 : scala.reflect.runtime.universe.TypeTag[T]) : org.apache.spark.sql.types.StructType = { /* compiled code */ }
I'm not familiar with Scala and it appears that it's not possible to create TypeTag[Example] in java.
Appreciate if you could share your thoughts on the below
- Is it safe (and possible) to just consume
TensorFlowInferSchemain Java without going viaspark.read.format("tfrecord").option("recordType", "Example")? - What to pass for the
TypeTag[T]argument? - Is there a Java example for this use case
Thanks for maintaining this project and highly appreciate your help!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The entry point mentioned is TensorFlowInferSchema.apply, with the Java Spark snippet and compiled signature as the starting context. Review how the TypeTag parameter and spark.read.format("tfrecord") path are exposed, then document a verified Java usage example answering the three questions; done means the guidance is complete and runnable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark, tensorflow
- Domain
- data-engineering
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100