holdenk / holdenk/spark-testing-base

Comparing RDD of org.apache.spark.sql.Row containing byte[] as values fails for the same provided RDD

Open
#247 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

Hello!
There are some issues with the spark library test base when we are using byte[] values in org.apache.spark.sql.Row.
The below tests were tested on s both spark test versions: 1.6.0 and 2.3.0.

The following code will always fail:

import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.junit.Test;

import scala.Option;
import scala.Tuple3;

import com.holdenkarau.spark.testing.JavaRDDComparisons;
import com.holdenkarau.spark.testing.SharedJavaSparkContext;

public class SparkSQLRowValueBytesFailedTests extends SharedJavaSparkContext
implements Serializable {
@Test
public void testSQLRowWithValueBytesFailes() throws Exception {

	Row row1 = RowFactory
			.create(new Object[] { "bytes".getBytes(), "good" });// it fails

	List<Row> rows = new ArrayList<>();

	rows.add(row1);
	
	JavaRDD<Row> expected = jsc().parallelize(rows);

	Option<Tuple3<Row, Integer, Integer>> diff2 = JavaRDDComparisons
			.compareRDD(expected, expected);
	
	//This fails		
	assertEquals(scala.None$.MODULE$, diff2);

}

//While this test passes

@Test
public void testSQLRowWithValueString() throws Exception {

	Row row1 = RowFactory
			.create(new Object[] { "bytes", "good" });// it fails

	List<Row> rows = new ArrayList<>();

	rows.add(row1);
	// rows.add(row2);

	JavaRDD<Row> expected = jsc().parallelize(rows);

	Option<Tuple3<Row, Integer, Integer>> diff2= JavaRDDComparisons
			.compareRDD(expected, expected);

//this passes
assertEquals(scala.None$.MODULE$, diff2);

}

}

junit-failed

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at JavaRDDComparisons.compareRDD and reproduce the provided Spark SQL Row cases with byte[] and String values. Compare the returned diff2 values and the failing assertEquals expectation; done means comparing the same RDD containing byte[] values returns scala.None$.MODULE$ as it does for String values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.