The spark3.x local mode is unable to read data from geomesa

The spark3.x local mode is unable to read data from geomesa

Spark3.x is not able to read the data of geomesa， but spark 2.x can.

Spark Version: 3.0.2
Geomesa Hbase Version: 3.2.0
Spark Deployed Mode: local
Operating System: Linux（CentOS7）

There is no problem changing the operating system to windows11 and running in IDEA, but the data cannot be read under Linux.

---

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-3116

Key: GEOMESA-3116
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: hujian
Created: Tue, 3 Aug 2021 09:30:38 -0400
Updated: Thu, 5 Aug 2021 11:10:00 -0400
Affects Version(s): 3.2.0
Component(s): HBase
JIRA Labels: geomesa_spark_sql

---

Attachments:

• image-20210804-071645.png (39.7 KB) - uploaded by ug:072a92e1-b061-4900-9ace-5c9b7225b285 on Wed, 4 Aug 2021 03:24:46 -0400

• image-20210804-072020.png (51.6 KB) - uploaded by ug:072a92e1-b061-4900-9ace-5c9b7225b285 on Wed, 4 Aug 2021 03:24:46 -0400

---

Comments:

557058:48912df4-0004-4f90-9915-db26684ec006 (Tue, 3 Aug 2021 14:10:05 -0400):

Hi hujian can you post the error that you got when trying to use Spark 3.x?

6109396c627b5600680eb942 (Wed, 4 Aug 2021 03:24:46 -0400):

Hi James Hughes ,

Code：
object Test2 { private val logger = LoggerFactory.getLogger("Test2") def main(args: Array[String]): Unit = { System.setProperty("HADOOP_USER_NAME", "hdfs") val catalog = "test" val sftName = "hj" val spark = SparkSession .builder() .appName("Test2") .master("local[*]") .getOrCreate() val dsParams = Map( "hbase.catalog" -> catalog, "hbase.zookeepers" -> "zookeeper1.local:2181,zookeeper2.local:2181,zookeeper3.local:2181" ) val df = spark.read .format("geomesa") .options(dsParams) .option("geomesa.feature", sftName) .load() logger.error("==============count: " + df.count()) df.show() spark.stop() } }
The correct result in Windows 11 is as follows（There are 6 rows of data in total）：

21/08/04 14:32:57 ERROR Test2: ==============count: 6

---

 fid int1 double2 string3  time4    

---

66cb069f-c21d-474...  3  2.34 五六七3 2021-06-27 22:54:27   b8681fd6-778d-48e...  4  2.34 五六七4 2021-06-27 22:54:27   1500c3a2-8d39-432...  1  2.34  五六七 2021-06-27 22:54:27   19764aae-ec19-4c8...  9  2.34 五六七9 2021-06-27 22:54:27   1e828229-0fd7-4e5...  2  2.34 五六七2 2021-06-27 22:54:27   f3240b61-8554-474...  5  2.34 五六七5 2021-06-27 22:54:27    

---

In Linux, there is no exception information, but the df.count() operation is being executed all the time，But in windows 11, the same operation can get the result in a few seconds.

Spark UI：

When I remove the .master("local*") and change to Spark on YARN Client / Cluster mode, I can also quickly get the correct results.

No repository files or tests are named. Start with the provided Scala Test2 example and compare Spark local[*] on Linux with Windows and YARN, focusing on the hanging df.count() call; done means the Linux local-mode read completes and returns the six expected rows.