Many threads blocked in LRUMap (with `ScalaObjectMapper`) -- increase type cache size?

(NOTE: moved from `jackson-databind` on 11-Sep-2019 by @cowtowncoder )

Version: Jackson 2.8.11, probably also 2.9.7

We have an API server that receives requests then makes ElasticSearch requests using the elastic4s library. The JSON parsing is done using Jackson under the hood. We’ve noticed a lot of threads blocked on the synchronized blocks in the `LRUMap` data structure, which seems to be contributing to slow performance on our API server.

Several threads are blocked on puts to the map, for example:

```
java.lang.Thread.State: BLOCKED (on object monitor)
at com.fasterxml.jackson.databind.util.LRUMap.putIfAbsent(LRUMap.java:58)
- waiting to lock <0x0000000629be2270> (a com.fasterxml.jackson.databind.util.LRUMap)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1287)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory.constructMapLikeType(TypeFactory.java:807)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:68)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$anonfun$constructType$1(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$$Lambda$2573/1098099028.apply(Unknown Source)
at scala.collection.immutable.List.map(List.scala:286)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$anonfun$constructType$1(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$$Lambda$2573/1098099028.apply(Unknown Source)
at scala.collection.immutable.List.map(List.scala:286)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$anonfun$constructType$1(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$$Lambda$2573/1098099028.apply(Unknown Source)
at scala.collection.immutable.List.map(List.scala:286)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:190)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:189)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.readValue(JacksonSupport.scala:11)
at com.sksamuel.elastic4s.http.ResponseHandler$.fromEntity(ResponseHandler.scala:34)
at models.es.ESHttpClientBackend$GetMappingHttpExecutable$$anon$1.handle(ESHttpClientBackend.scala:575)
at com.sksamuel.elastic4s.http.HttpClient.$anonfun$execute$1(HttpClient.scala:62)
at com.sksamuel.elastic4s.http.HttpClient$$Lambda$2539/1265612311.apply(Unknown Source)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.Future$$Lambda$1146/1816269091.apply(Unknown Source)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.Promise$$Lambda$609/1310865036.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```

and another thread blocked on `clear()`:

```
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)
- locked <0x0000000625710000> (a java.util.concurrent.ConcurrentHashMap$TreeBin)
at com.fasterxml.jackson.databind.util.LRUMap.clear(LRUMap.java:70)
at com.fasterxml.jackson.databind.util.LRUMap.putIfAbsent(LRUMap.java:60)
- locked <0x0000000629be2270> (a com.fasterxml.jackson.databind.util.LRUMap)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1287)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1396)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory._fromParamType(TypeFactory.java:1400)
at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1165)
at com.fasterxml.jackson.databind.type.TypeFactory._resolveSuperInterfaces(TypeFactory.java:1311)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1254)
at com.fasterxml.jackson.databind.type.TypeFactory.constructMapLikeType(TypeFactory.java:807)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:68)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$anonfun$constructType$1(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$$Lambda$2573/1098099028.apply(Unknown Source)
at scala.collection.immutable.List.map(List.scala:286)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType(ScalaObjectMapper.scala:64)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.constructType$(ScalaObjectMapper.scala:56)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.constructType(JacksonSupport.scala:11)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:190)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:189)
at com.sksamuel.elastic4s.json.JacksonSupport$$anon$1.readValue(JacksonSupport.scala:11)
at com.sksamuel.elastic4s.http.ResponseHandler$.fromEntity(ResponseHandler.scala:34)
at models.es.ESHttpClientBackend$GetMappingHttpExecutable$$anon$1.handle(ESHttpClientBackend.scala:575)
at com.sksamuel.elastic4s.http.HttpClient.$anonfun$execute$1(HttpClient.scala:62)
at com.sksamuel.elastic4s.http.HttpClient$$Lambda$2539/1265612311.apply(Unknown Source)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.Future$$Lambda$1146/1816269091.apply(Unknown Source)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.Promise$$Lambda$609/1310865036.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```

Based on heap dumps, we see that the `_typeCache` map in `TypeFactory` is getting filled up quickly with type information for the JSON objects we’re parsing, then the map is getting cleared, then the process is repeated quickly. Since the `clear()` operation is relatively expensive and inside a synchronized block, it blocks all other threads from making progress, and this can happen repeatedly in a short period of time.

We could replace the `ConcurrentHashMap` with a synchronized `LinkedHashMap` with an LRU eviction policy, and/or increase the size of the cache. Either way, clearing the cache seems like an inefficient solution, especially when many threads are trying to use the cache.