CPU goes crazy in ignite server

Greetings,

Our system works well before but recently CPU goes high and never get down.
The architecture is like that, we have three clients, five servers whose CPU have 16 cores per each.
The CPU of clients look good but the five servers are in very high cpu%.
After checking the thread dump, we can see all busy threads are from Query Thread Pool with prefix "query-#".

e.g.

"query-#115" #174 prio=5 os_prio=0 tid=0x000055c49d171000 nid=0x14b runnable [0x00007f30fb564000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringCoding.decode(StringCoding.java:229)
at java.lang.String.(String.java:463)
at org.apache.ignite.internal.binary.BinaryObjectImpl.fieldByOrder(BinaryObjectImpl.java:428)
at org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:112)
at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:223)
at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:120)
at org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:235)
at org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue0(H2CacheRow.java:107)
at org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue(H2CacheRow.java:91)
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:479)
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:72)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5377)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.findLowerBound(BPlusTree.java:5494)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5767)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5607)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next(BPlusTree.java:5840)
at org.apache.ignite.internal.processors.query.h2.H2Cursor.next(H2Cursor.java:66)
at org.h2.index.IndexCursor.next(IndexCursor.java:316)
at org.h2.table.TableFilter.next(TableFilter.java:502)
at org.h2.table.TableFilter.next(TableFilter.java:541)
at org.h2.table.TableFilter.next(TableFilter.java:541)
at org.h2.command.dml.Select.queryGroup(Select.java:323)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:628)
at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114)
at org.h2.command.dml.Query.query(Query.java:352)
at org.h2.command.dml.Query.query(Query.java:333)
at org.h2.command.CommandContainer.query(CommandContainer.java:114)
at org.h2.command.Command.executeQuery(Command.java:202)
- locked <0x00007f3545d078b0> (a org.h2.engine.Session)
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
- locked <0x00007f3545d078b0> (a org.h2.engine.Session)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:821)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:914)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:407)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2171)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2124)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$379/2097614581.onMessage(Unknown Source)
at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3477)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1907)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1528)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:241)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1421)
at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

our code is simplified like this: 100 requests per seconds

public void save(Order Order) {
order.setLastModified(new Timestamp(System.currentTimeMillis()));
this.getCache().put(workingOrder.getOrderId(), workingOrder);
}

My questions are:
1. where can I check SQL histories of the server? I checked the view SQL_QUERIES_HISTORY, but it only contains the SQLs executed from the connected client.
2. how we can troubleshoot such problem? what kind of metrics we can use? we run pure ignite server referring to https://ignite.apache.org/docs/2.9.1/installation/installing-using-docker

Any idea appreciated.

Start with the supplied thread dump, especially BinaryObjectImpl.fieldByOrder, H2Tree.compare, IgniteH2Indexing.executeSqlQuery, and GridMapQueryExecutor.onQueryRequest0. Review SQL_QUERIES_HISTORY and the Ignite 2.9.1 Docker installation context, then identify which server-side queries or metrics account for the query-thread CPU; done requires a reproducible diagnosis and documented troubleshooting evidence.

Written by the indexing model from the issue text.