apache / apache/gravitino

[Bug report] Dropping a MySQL-backed catalog leaks its ClassLoader via AbandonedConnectionCleanupThread and non-webserver ThreadLocals

Open
#12,123 0 comments 0 reactions 2 assignees Claimed by @yuqi1129 View on GitHub
1.3.1 bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Version

main branch

### Describe what's wrong

When a catalog that used the MySQL Connector/J driver is dropped or reloaded, its isolated ClassLoader is not garbage collected, so Metaspace grows on every reload and eventually hits `OutOfMemoryError: Metaspace`.

Two references keep the ClassLoader alive after cleanup runs:

1. MySQL's `AbandonedConnectionCleanupThread` (`mysql-cj-abandoned-connection-cleanup`) holds the driver's ClassLoader as its context ClassLoader. A plain thread interrupt does not release it; only `uncheckedShutdown()` clears the tracked-connection map so the thread and ClassLoader can be collected.
2. `ClassLoaderResourceCleanerUtils.clearThreadLocalMap` only sweeps `Gravitino-webserver-*` threads. ThreadLocals that reference the dropped ClassLoader can also sit on other threads (Caffeine ForkJoinPool workers, catalog-cleaner, Hadoop daemons), and those are never cleared.

`closeClassLoaderResource` is the shared routine that releases a catalog ClassLoader's resources, but it releases neither of the above references, so a MySQL-backed catalog leaks its ClassLoader whenever cleanup runs.

### Error message and/or stacktrace

`java.lang.OutOfMemoryError: Metaspace` after repeatedly creating and dropping MySQL-backed catalogs (Metaspace committed size grows monotonically with reload count).

### How to reproduce

1. Gravitino version: main branch
2. Create a catalog that loads the MySQL driver (a `jdbc-mysql` catalog, or an Iceberg JDBC catalog on MySQL).
3. Drop the catalog, then repeat create/drop in a loop.
4. Observe with `jcmd VM.metaspace` (or a heap dump) that the isolated ClassLoader and the `mysql-cj-abandoned-connection-cleanup` thread survive each drop; Metaspace committed size keeps climbing.

### Additional context

The fix is scoped to `ClassLoaderResourceCleanerUtils`: shut down the MySQL `AbandonedConnectionCleanupThread` (guarded so it only acts when the target ClassLoader owns the driver class), and broaden the ThreadLocal sweep beyond webserver threads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.