H22 cache sync fallback blocks startup indefinitely on the MVStore fair lock

Problem

An instance cannot complete startup. The main thread is parked indefinitely inside an H22 (H2 on-disk cache) write, driven from InitServlet.init():

"main" ... waiting on condition
   - parking to wait for <0x00000007fe4a3718> (a java.util.concurrent.locks.ReentrantLock$FairSync)
	at org.h2.mvstore.MVStore.commit(MVStore.java:775)
	at org.h2.mvstore.MVStore.beforeWrite(MVStore.java:1273)
	...
	at com.dotmarketing.business.cache.provider.h22.H22Cache.doUpsert(H22Cache.java:650)
	at com.dotmarketing.business.cache.provider.h22.H22Cache.put(H22Cache.java:176)
	...
	at com.dotcms.content.elasticsearch.business.ESContentFactoryImpl.lambda$findContentlets$0(ESContentFactoryImpl.java:1306)
	at com.dotcms.vanityurl.business.VanityUrlAPIImpl.populateAllVanityURLsCache(VanityUrlAPIImpl.java:128)
	at com.dotmarketing.servlets.InitServlet.init(InitServlet.java:113)

Startup never finishes. A cache write should never be able to block startup.

Root cause

H22Cache.java:176 is the synchronous branch of put(). shouldAsync() returned false, so the caller performs the H2 write itself. Nothing bounds that write:

• MVStore.commit takes one fair ReentrantLock per store (confirmed by ReentrantLock$FairSync in the dump). Embedded H2 is single-writer; fair means strict FIFO with no barging.
• cache.h22.db.poolsize.max defaults to 500, so up to 500 threads can hold a connection and queue on that single lock.
• Hikari's connectionTimeout=1000 bounds acquiring a connection, not executing one. No setQueryTimeout is set on any H22 statement, so the wait is unbounded.

The overflow valve is what causes the stall

isAllocationWithinTolerance() flips at inFlightTasks >= 0.98 x 10000 = 9800. Past that threshold, shouldAsync() returns false and every caller becomes a writer. Concurrency goes from 5 (capped by dbWorkPermits) to potentially hundreds, all queued on one fair lock. Throughput collapses, the backlog never drains, and the tolerance never clears. It latches instead of shedding load.

ESContentFactoryImpl:1305-1308 (result.forEach(... contentletCache.add ...), 200 rows per batch, no pacing) reaches that threshold quickly on a site with many vanity URLs.

cache_h22_async_task_queue no longer describes anything

Since #35992 moved async commits to Executors.newThreadPerTaskExecutor, there is no bounded queue. The real concurrency bound is dbWorkPermits = cache_h22_async_threads (default 5). cache_h22_async_task_queue=10000 is now only the threshold at which the provider flips to caller-runs. Meanwhile 9800 parked virtual threads each pin their serialized value, adding heap pressure.

Fix

1. In async mode, never fall back to running the write on the caller. On overflow, drop the write. This is already safe and is the existing error semantic: put() marks the key in DONT_CACHE_ME (H22Cache.java:167) before writing, doUpsert only clears it on success (:655), and doSelect honors exclude() (:664) — so a dropped put reads as a miss. Caller-runs remains only when cache_h22_async=false is set explicitly.
2. Bound overflow with the permit itself — dbWorkPermits.tryAcquire(timeout) and drop on failure. This replaces the phantom queue/tolerance calculation, which can then be deleted.
3. setQueryTimeout on H22 statements as a backstop so a wedged statement eventually fails instead of parking forever.

Puts are droppable, deletes are not

DONT_CACHE_ME expires after 20s (H22Cache.java:73). A dropped put is safe indefinitely — after expiry there is simply no row to find. A dropped delete is not: after 20s the stale row becomes visible again and serves stale content. So deletes must still block on the permit (on a virtual thread, never on the caller).

This also means submitAsync's current RejectedExecutionException handling is a latent stale-read bug, since it silently drops deletes as well as puts.

Immediate mitigation (config only, no build)

DOT_CACHE_H22_DB_POOLSIZE_MAX=10

Caps the fair-lock queue. The 11th caller hits connectionTimeout and fails in ~1s into handleError, so startup proceeds with degraded caching instead of hanging.

Or drop the H22 tier entirely — it is a disk cache, so this costs read performance, not correctness:

DOT_CACHE_CONTENTLETCACHE_CHAIN=com.dotmarketing.business.cache.provider.caffine.CaffineCache

Acceptance criteria

• With cache_h22_async=true, no caller thread ever executes doUpsert/doDelete; verified by the absence of a synchronous path from put()/remove().
• Saturating the async path sheds puts (counted/logged) rather than running them on the caller, and startup completes.
• Deletes are never silently dropped while the provider is running.
• H22 statements carry a query timeout.
• cache_h22_async_task_queue / cache_h22_async_tolerance are removed, along with the already-@Ignored Test_Exhaust_Thread_Pool that covered them.

Start in H22Cache.java at put(), remove(), submitAsync(), doUpsert(), and doDelete(), then trace the startup path through ESContentFactoryImpl and InitServlet.init(). Review the ignored Test_Exhaust_Thread_Pool and the H22 statement setup, and run the relevant cache tests. Done means async callers never execute writes, puts shed load safely, deletes are not silently dropped, statements have timeouts, and obsolete queue settings and test code are removed.