Ignite semaphore does not perform well at high concurrency. The API contract of tryAcquire is also violated and the method does not return within the supplied timeout . Ignite 2.16.0

We had need to use a distributed semaphore to control access to certain resources. eg. we need to limit number of concurrent downloads / uploads etc. We recently switched to a ignite semaphore because we like the property that the permits are release if a connection is lost. Our prior solution did not have this property and permits were lost when out server went down.
However, the problem with the ignite semaphore was that it does not work well when 100 threads hit it at a time. We also tried the tryAcquire(1, 100, TimeUnit.MILLISECONDS). We noticed that it started waiting for 10, 15, 20 secs (whether permits are available or not). When the number of threads is low the API works as expected.

I have provided a sample program which simulates the problem. The P95 on my system was around 6 sec. When i am expecting the method to return in 100ms all the time. Our production setup is usually a 3 node ingite nodes. 3 zookeeper nodes and 3 platform nodes (our tomcat process) . However to simplify i am running zookeeper and ignite on the same node.

Start with the supplied sample program and the distributed semaphore's tryAcquire(1, 100, TimeUnit.MILLISECONDS) behavior under roughly 100 concurrent threads. Reproduce the reported latency with the simplified Ignite and ZooKeeper setup, then verify that tryAcquire returns within the supplied timeout while preserving semaphore behavior when permits or connections change.

Written by the indexing model from the issue text.