holdenk / holdenk/spark-testing-base
Comparing RDD of org.apache.spark.sql.Row containing byte[] as values fails for the same provided RDD
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
Hello!
There are some issues with the spark library test base when we are using byte[] values in org.apache.spark.sql.Row.
The below tests were tested on s both spark test versions: 1.6.0 and 2.3.0.
The following code will always fail:
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.junit.Test;
import scala.Option;
import scala.Tuple3;
import com.holdenkarau.spark.testing.JavaRDDComparisons;
import com.holdenkarau.spark.testing.SharedJavaSparkContext;
public class SparkSQLRowValueBytesFailedTests extends SharedJavaSparkContext
implements Serializable {
@Test
public void testSQLRowWithValueBytesFailes() throws Exception {
Row row1 = RowFactory
.create(new Object[] { "bytes".getBytes(), "good" });// it fails
List<Row> rows = new ArrayList<>();
rows.add(row1);
JavaRDD<Row> expected = jsc().parallelize(rows);
Option<Tuple3<Row, Integer, Integer>> diff2 = JavaRDDComparisons
.compareRDD(expected, expected);
//This fails
assertEquals(scala.None$.MODULE$, diff2);
}
//While this test passes
@Test
public void testSQLRowWithValueString() throws Exception {
Row row1 = RowFactory
.create(new Object[] { "bytes", "good" });// it fails
List<Row> rows = new ArrayList<>();
rows.add(row1);
// rows.add(row2);
JavaRDD<Row> expected = jsc().parallelize(rows);
Option<Tuple3<Row, Integer, Integer>> diff2= JavaRDDComparisons
.compareRDD(expected, expected);
//this passes
assertEquals(scala.None$.MODULE$, diff2);
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at JavaRDDComparisons.compareRDD and reproduce the provided Spark SQL Row cases with byte[] and String values. Compare the returned diff2 values and the failing assertEquals expectation; done means comparing the same RDD containing byte[] values returns scala.None$.MODULE$ as it does for String values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100