eclipse-ee4j / eclipse-ee4j/eclipselink

Configuration of CACHE_QUERY_FORCE_DEFERRED_LOCKS in org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader) taking place too late

Open
#1,920 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
246
Forks
202
Avg merge
1d 22h
Merged PRs (30d)
14

Description

Please be aware,

We currently have a tracker open in oracle related to dead locks in production due to a bug in the:
org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader)

In eclipse link 2.7.6 there was the problem that the line of code:

```
String queryCacheForceDeferredLocks = getConfigPropertyAsStringLogDebug(PersistenceUnitProperties.CACHE_QUERY_FORCE_DEFERRED_LOCKS, m, session);
session.getProject().setQueryCacheForceDeferredLocks("true".equalsIgnoreCase(queryCacheForceDeferredLocks));
```

Was missing.
But for this bug there are official oracle patches.

But then all subsequent version, including the
https://github.com/eclipse-ee4j/eclipselink/blob/2.7.13/jpa/org.eclipse.persistence.jpa/src/org/eclipse/persistence/internal/jpa/EntityManagerSetupImpl.java

There is the problem that the line of code is placed too late.

The line code is being placed after:
```
Map typeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_TYPE_, m, session);
Map sizeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SIZE_, m, session);
Map sharedMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SHARED_, m, session);
if(typeMap.isEmpty() && sizeMap.isEmpty() && sharedMap.isEmpty()) {
return;
}
```

And this a gigantic problem because as it turns out, the weblogic deloyments seem to be coming out on that line code.

Our persistence.xml had the following configuration properties to mitigate the risk of dead locks:
```





















```

But the holy grail of the config properties:

Was not doing anything.

So we had the typical dead locks with some threads waiting to release deferred locks.
Those are the nice threads that be having properly.

```
java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(java.base@11.0.13/Native Method)

at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:653)

at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1107)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInternal(ObjectBuilder.java:774)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:727)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:705)

at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:861)

at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:570)

at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)

at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)

at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)

at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:447)

at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:3367)
```


So this part was looking good in the dead locks. These are the threads we know that cause no trouble.

But then we had the nasty ones;

```

"[STUCK] ExecuteThread: '69' for queue: 'weblogic.kernel.Default (self-tuning)'" #366 daemon prio=1 os_prio=0 cpu=11193.21ms elapsed=1815.58s tid=0x00007ff6e405f000 nid=0x37686 in Object.wait() [0x00007ff395595000]

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.13/Native Method)

- waiting on

at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:128)

- waiting to re-lock in wait() <0x0000000309d45180> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)

at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:139)

at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:110)

at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:175)

at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:101)

at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.acquireLock(IsolatedClientSessionIdentityMapAccessor.java:78)

at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:92) <------- Tthis is a complete application distructor code line, the retrieveCacheKey should be calling for the
----- cacheKey = this.getIdentityMapAccessorInstance().acquireDeferredLock(primaryKey, concreteDescriptor.getJavaClass(), concreteDescriptor, query.isCacheCheckComplete() || query.shouldRetrieveBypassCache());


at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5383) <-----Was choosing the wrong code path
```

So please be aware of this defect.

I would also very strongly recommend, please make the eclipselink default have the force deferred locks active to TRUE and put in capital letters a strong recommendation against ever setting it to false.
The constructor of the:
org.eclipse.persistence.queries.ObjectBuildingQuery.ObjectBuildingQuery()

Should probably default the

` this.requiresDeferredLocks = true;`

and finally the

org.eclipse.persistence.queries.ObjectBuildingQuery.requiresDeferredLocks()

In case of null should return true.

Thanks for considering these suggestions

Contributor guide

Open the contributing guide

Research direction

Start with EntityManagerSetupImpl.java, especially updateDescriptorCacheSettings(Map, ClassLoader), and compare when CACHE_QUERY_FORCE_DEFERRED_LOCKS is applied with the cache maps and early return. Then inspect ObjectBuildingQuery.ObjectBuildingQuery() and requiresDeferredLocks(); done means the configured property is applied before lock acquisition and the requested default behavior is consistent with the documented setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.