[Bug report] Kerberos TGT refresh threads leak in Iceberg REST server due to CGLIB double-construction in KerberosAwareIcebergCatalogProxy
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### Version
main branch
### Describe what's wrong
Describe the bug
When Kerberos authentication is enabled for the Iceberg REST catalog, the process accumulates a large number of idle daemon threads named check-Iceberg-Hive-tgt-0 (or similar ticket-refresh threads). These threads never exit and grow roughly in sync with catalog-wrapper cache recreation (default eviction interval: 1 hour).
In a long-running production instance (~30 days), a thread dump showed ~1000 threads, most of which were stuck in:
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
Each thread waited on a different Condition object, which indicates many separate ScheduledThreadPoolExecutor instances were created and never shut down.
Root cause
IcebergCatalogWrapperManager.createCatalogWrapper() does:
new CatalogWrapperForREST(...) → creates catalog A + KerberosClient A + TGT refresh thread A
new KerberosAwareIcebergCatalogProxy(rest).getProxy(catalogName, config) → CGLIB Enhancer.create(constructorArgs) re-invokes the constructor → creates catalog B + KerberosClient B + TGT refresh thread B
The cache only retains the proxy. On eviction, proxy.close() is intercepted and delegated to target.close(), which closes A only. B is never closed, so its scheduler thread leaks forever.
Relevant code:
iceberg/.../KerberosAwareIcebergCatalogProxy.java — e.create(argClass, new Object[]{...}) re-runs constructors
iceberg/.../authentication/kerberos/KerberosClient.java — creates ScheduledThreadPoolExecutor named check-Iceberg-Hive-tgt-%d
iceberg-rest-server/.../IcebergCatalogWrapperManager.java — expireAfterWrite cache recreates wrappers periodically
Note: migrating to the shared catalogs/hadoop-auth KerberosClient improves per-client lifecycle (e.g. shutdown before re-login), but does not fix this leak, because the orphaned client from CGLIB construction is still never closed.
Expected behavior
Each Kerberos-enabled catalog wrapper lifecycle should own exactly one KerberosClient / TGT refresh executor. When the wrapper is closed (including cache eviction), that executor must be shut down and the thread must exit. Recreating the wrapper must not leave orphaned refresh threads behind.
Suggested fix
Instantiate the CGLIB proxy without invoking IcebergCatalogWrapper / CatalogWrapperForREST constructors (e.g. via Objenesis + Factory.setCallback), and keep all method calls delegated to the already-constructed target.
Avoid “fix only in close()” as the primary solution: the proxy should never own a second catalog/Kerberos client in the first place.
### Error message and/or stacktrace
"check-Iceberg-Hive-tgt-0" #17526 daemon prio=5 os_prio=0 cpu=140.41ms elapsed=2117176.03s tid=0x00007fe07ad9dab0 nid=0xf2d10 waiting on condition [0x00007fdce77f6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
- parking to wait for <0x00000800b5ab4d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.2/LockSupport.java:252)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.2/AbstractQueuedSynchronizer.java:1672)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.2/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.2/ThreadPoolExecutor.java:1062)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.2/ThreadPoolExecutor.java:1122)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.2/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
### How to reproduce
1.1.0
### Additional context
_No response_
Contributor guide
Research direction
Start in iceberg/.../KerberosAwareIcebergCatalogProxy.java and inspect how e.create invokes constructors, then trace wrapper creation and eviction in iceberg-rest-server/.../IcebergCatalogWrapperManager.java. Use iceberg/.../authentication/kerberos/KerberosClient.java to understand the refresh executor; done means each catalog wrapper owns one client and cache eviction leaves no orphaned TGT refresh thread.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- authentication, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100