Investigate caching at the datastore level

Investigate caching at the datastore level

Currently we cache simple feature types, etc fairly aggressively. Possibly we should tie our caching to the datastore lifecycle. However, this may increase the time to create a new data store, which may negatively impact spark operations (e.g. see our issues with HBase connection instances). The impetus for this is that tomcat complains on undeploy:
SEVERE: The web application [geoserver] created a ThreadLocal with key of type [org.locationtech.geomesa.utils.cache.SoftThreadLocalCache$$anon$1] (value [org.locationtech.geomesa.utils.cache.SoftThreadLocalCache$$anon$1@71540204]) and a value of type [scala.collection.mutable.HashMap] (value [Map(tts:layert;date:Date,a:String,i:Integer,c:Boolean,geom:Point,var:String -> scala.ref.SoftReferenceWithWrapper@386653a7)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2441

Key: GEOMESA-2441
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Fri, 19 Oct 2018 09:45:18 -0400
Updated: Fri, 19 Oct 2018 09:45:18 -0400

Start by tracing SoftThreadLocalCache and how datastore creation and shutdown are handled. Review the noted HBase connection-instance impact on Spark operations and the Tomcat undeploy warning. Done should include a decided caching lifecycle approach that avoids the reported ThreadLocal leak without unacceptable datastore creation costs.