eclipse-ee4j / eclipse-ee4j/eclipselink
Configuration of CACHE_QUERY_FORCE_DEFERRED_LOCKS in org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader) taking place too late
- Dominant language
- Java
- Stars
- 246
- Forks
- 202
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 14
Description
Please be aware,
We currently have a tracker open in oracle related to dead locks in production due to a bug in the:
org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader)
In eclipse link 2.7.6 there was the problem that the line of code:
```
String queryCacheForceDeferredLocks = getConfigPropertyAsStringLogDebug(PersistenceUnitProperties.CACHE_QUERY_FORCE_DEFERRED_LOCKS, m, session);
session.getProject().setQueryCacheForceDeferredLocks("true".equalsIgnoreCase(queryCacheForceDeferredLocks));
```
Was missing.
But for this bug there are official oracle patches.
But then all subsequent version, including the
https://github.com/eclipse-ee4j/eclipselink/blob/2.7.13/jpa/org.eclipse.persistence.jpa/src/org/eclipse/persistence/internal/jpa/EntityManagerSetupImpl.java
There is the problem that the line of code is placed too late.
The line code is being placed after:
```
Map typeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_TYPE_, m, session);
Map sizeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SIZE_, m, session);
Map sharedMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SHARED_, m, session);
if(typeMap.isEmpty() && sizeMap.isEmpty() && sharedMap.isEmpty()) {
return;
}
```
And this a gigantic problem because as it turns out, the weblogic deloyments seem to be coming out on that line code.
Our persistence.xml had the following configuration properties to mitigate the risk of dead locks:
```
```
But the holy grail of the config properties:
Was not doing anything.
So we had the typical dead locks with some threads waiting to release deferred locks.
Those are the nice threads that be having properly.
```
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.13/Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:653)
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1107)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInternal(ObjectBuilder.java:774)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:727)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:705)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:861)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:570)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:447)
at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:3367)
```
So this part was looking good in the dead locks. These are the threads we know that cause no trouble.
But then we had the nasty ones;
```
"[STUCK] ExecuteThread: '69' for queue: 'weblogic.kernel.Default (self-tuning)'" #366 daemon prio=1 os_prio=0 cpu=11193.21ms elapsed=1815.58s tid=0x00007ff6e405f000 nid=0x37686 in Object.wait() [0x00007ff395595000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.13/Native Method)
- waiting on
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:128)
- waiting to re-lock in wait() <0x0000000309d45180> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:139)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:110)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:175)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:101)
at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.acquireLock(IsolatedClientSessionIdentityMapAccessor.java:78)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:92) <------- Tthis is a complete application distructor code line, the retrieveCacheKey should be calling for the
----- cacheKey = this.getIdentityMapAccessorInstance().acquireDeferredLock(primaryKey, concreteDescriptor.getJavaClass(), concreteDescriptor, query.isCacheCheckComplete() || query.shouldRetrieveBypassCache());
at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5383) <-----Was choosing the wrong code path
```
So please be aware of this defect.
I would also very strongly recommend, please make the eclipselink default have the force deferred locks active to TRUE and put in capital letters a strong recommendation against ever setting it to false.
The constructor of the:
org.eclipse.persistence.queries.ObjectBuildingQuery.ObjectBuildingQuery()
Should probably default the
` this.requiresDeferredLocks = true;`
and finally the
org.eclipse.persistence.queries.ObjectBuildingQuery.requiresDeferredLocks()
In case of null should return true.
Thanks for considering these suggestions
Contributor guide
Research direction
Start with EntityManagerSetupImpl.java, especially updateDescriptorCacheSettings(Map, ClassLoader), and compare when CACHE_QUERY_FORCE_DEFERRED_LOCKS is applied with the cache maps and early return. Then inspect ObjectBuildingQuery.ObjectBuildingQuery() and requiresDeferredLocks(); done means the configured property is applied before lock acquisition and the requested default behavior is consistent with the documented setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100