Accumulo Monitor cannot handle the truth

A system with 200 tservers, and an incorrect table.context set, will start spamming the monitor with so many messages that we cannot keep the monitor up and running as it will run out of heap very quickly even with 16G of memory. This is with a 2.1 accumulo instance. In the 1.x versions the monitor we had no issues handling this volume of messages.

A jmap -histo just before it falls over shows characters arrays being the majority of the problem followed by ArrayTrie objects and byte arrays. The top 5 entries were:

instances : bytes : classname
466527 : 15277159512 : [C (java.base@11.0.15)
155314 : 638615624 : [Lorg.eclipse.jetty.util.ArrayTrie$Node;
988606 : 119096256 : [B (java.base@11.0.15)
1692806 : 54169792 : java.util.concurrent.locks.ReentrantLock$NonfairSync (java.base@11.0.15)
1848284 : 44358816 : java.lang.String (java.base@11.0.15)

I will have to say I was supprised to see many of the threads using glassfish in the jstack. That might be a somewhat expensive servlet infrastructure to use in the monitor.

java 11.0.15
CentOS 7.5

Start up a system with 150+ tservers, and a monitor with 16g ram and lots of data across many tables. Set the table context to something bogus and watch the monitor quickly run out of memory. I am sure we could create a smaller system to simulate this.

Start at the Accumulo Monitor entry point and reproduce the failure with 150+ tservers, a bogus table context, and substantial table data on Java 11. Use the reported jmap heap histogram and jstack output to identify the allocation growth; done means the monitor remains running under this workload without rapidly exhausting its heap.

Written by the indexing model from the issue text.