Spark - 'cache' option requires extra classpath dependencies

Spark - 'cache' option requires extra classpath dependencies

When using: spark.read.option("cache", "true")...
You get: IllegalArgumentException: Cache argument set to true but GeoCQEngineDataStore is not on the classpath

The geomesa-cqengine-datastore_2.11 module needs to be bundled in the various spark-runtime jars for the cache option to work. However, it has always been marked 'provided' - is there a reason for that? Possibly documenting the classpath setup would be sufficient.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2602

Key: GEOMESA-2602
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Wed, 10 Apr 2019 11:08:43 -0400
Updated: Wed, 10 Apr 2019 16:23:17 -0400
Affects Version(s): 2.0.0, 2.1.0, 2.0.1, 2.0.2, 2.2.0, 2.1.1, 2.3.0, 2.1.2, 2.2.1, 2.2.2, 2.1.3

Comments:

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Wed, 10 Apr 2019 16:23:17 -0400):

Adding the geomesa-cqengine-datastore_2.11 to the spark classpath did allow the functionality to work, seemingly. Although large queries died, likely due to memory exhaustion.

Inspect the dependency declarations for the various spark-runtime jars and the geomesa-cqengine-datastore_2.11 module. Reproduce spark.read.option("cache", "true") and verify whether the required class is available; done means the cache option works with the runtime jars or the required classpath setup is clearly documented.