linkedin / linkedin/spark-tfrecord

Consume TensorFlowInferSchema from Java Spark 2.4.0

Open
#34 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
300
Forks
57
PR merge metrics
No merged PRs in 30d

Description

This is not an issue and is more of a request for guidance. I'd like to just use the TensorFlowInferSchema functionality in my Java spark job. Below is the sample snippet

JavaRDD<Example> exampleRdd =
          jsc.parallelize(
              Arrays.asList(
                  Example.newBuilder()
                      .setFeatures(
                          Features.newBuilder()
                              .putFeature(
                                  "Test",
                                  Feature.newBuilder()
                                      .setInt64List(Int64List.newBuilder().addValue(10).build())
                                      .build())
                              .build())
                      .build()));
StructType schema = TensorFlowInferSchema.apply(rdd.rdd(), ???);

However, the apply method expects a second argument - implicit evidence$1 : scala.reflect.runtime.universe.TypeTag[T].

Below is the compiled Java class

def apply[T](rdd : org.apache.spark.rdd.RDD[T])(implicit evidence$1 : scala.reflect.runtime.universe.TypeTag[T]) : org.apache.spark.sql.types.StructType = { /* compiled code */ }

I'm not familiar with Scala and it appears that it's not possible to create TypeTag[Example] in java.

Appreciate if you could share your thoughts on the below

  • Is it safe (and possible) to just consume TensorFlowInferSchema in Java without going via spark.read.format("tfrecord").option("recordType", "Example")?
  • What to pass for the TypeTag[T] argument?
  • Is there a Java example for this use case

Thanks for maintaining this project and highly appreciate your help!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The entry point mentioned is TensorFlowInferSchema.apply, with the Java Spark snippet and compiled signature as the starting context. Review how the TypeTag parameter and spark.read.format("tfrecord") path are exposed, then document a verified Java usage example answering the three questions; done means the guidance is complete and runnable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark, tensorflow
Domain
data-engineering
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.