Spatial Partitioning: errors interpreting sql expression

Spatial Partitioning: errors interpreting sql expression

SparkSQLData2Test.scala The join condition in case of spatial partitioning enabled works fine only if the tables and the related parameters in the join condition have the same order.

Moreover if the the parameters of the join condition are not included in the select fields the following exception is thown:

java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.spark.sql.execution.datasources.csv.UnivocityGenerator.convertRow(UnivocityGenerator.scala:66)
at org.apache.spark.sql.execution.datasources.csv.UnivocityGenerator.write(UnivocityGenerator.scala:82)
at org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.write(CSVFileFormat.scala:153)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:392)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:269)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:267)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1414)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:272)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

I attach a test case class for reproducing the described behaviors

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2730

Key: GEOMESA-2730
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Rodolfo Totaro
Created: Wed, 23 Oct 2019 14:46:14 -0400
Updated: Wed, 23 Oct 2019 15:21:17 -0400
Affects Version(s): 2.4.0, 2.3.1
Component(s): Converters/ETL
Parent: GEOMESA-1569
JIRA Labels: spark

Attachments:

• SparkSQLData2Test.scala (6.0 KB) - uploaded by rtotaro on Wed, 23 Oct 2019 15:11:25 -0400

Start with the attached SparkSQLData2Test.scala reproduction and run its spatial-partitioning cases. Trace the SQL join condition handling and the CSV output path implicated by UnivocityGenerator.convertRow. Done means joins work regardless of table or parameter order and do not fail when join parameters are omitted from the selected fields.