[Bug report] Dropping a MySQL-backed catalog leaks its ClassLoader via AbandonedConnectionCleanupThread and non-webserver ThreadLocals
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Version
main branch
### Describe what's wrong
When a catalog that used the MySQL Connector/J driver is dropped or reloaded, its isolated ClassLoader is not garbage collected, so Metaspace grows on every reload and eventually hits `OutOfMemoryError: Metaspace`.
Two references keep the ClassLoader alive after cleanup runs:
1. MySQL's `AbandonedConnectionCleanupThread` (`mysql-cj-abandoned-connection-cleanup`) holds the driver's ClassLoader as its context ClassLoader. A plain thread interrupt does not release it; only `uncheckedShutdown()` clears the tracked-connection map so the thread and ClassLoader can be collected.
2. `ClassLoaderResourceCleanerUtils.clearThreadLocalMap` only sweeps `Gravitino-webserver-*` threads. ThreadLocals that reference the dropped ClassLoader can also sit on other threads (Caffeine ForkJoinPool workers, catalog-cleaner, Hadoop daemons), and those are never cleared.
`closeClassLoaderResource` is the shared routine that releases a catalog ClassLoader's resources, but it releases neither of the above references, so a MySQL-backed catalog leaks its ClassLoader whenever cleanup runs.
### Error message and/or stacktrace
`java.lang.OutOfMemoryError: Metaspace` after repeatedly creating and dropping MySQL-backed catalogs (Metaspace committed size grows monotonically with reload count).
### How to reproduce
1. Gravitino version: main branch
2. Create a catalog that loads the MySQL driver (a `jdbc-mysql` catalog, or an Iceberg JDBC catalog on MySQL).
3. Drop the catalog, then repeat create/drop in a loop.
4. Observe with `jcmd VM.metaspace` (or a heap dump) that the isolated ClassLoader and the `mysql-cj-abandoned-connection-cleanup` thread survive each drop; Metaspace committed size keeps climbing.
### Additional context
The fix is scoped to `ClassLoaderResourceCleanerUtils`: shut down the MySQL `AbandonedConnectionCleanupThread` (guarded so it only acts when the target ClassLoader owns the driver class), and broaden the ThreadLocal sweep beyond webserver threads.
Contributor guide
Assessment
This issue has not been assessed yet.