apache / apache/gravitino

[Improvement] Glue catalog does not release its ClassLoader: an AWS client is left open, keeping the SDK's idle-connection reaper alive

Open
#13,016 0 comments 0 reactions 1 assignee Claimed by @diqiu50 View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

The Glue catalog does not release its isolated ClassLoader when it is dropped, so its classes stay in Metaspace for the life of the process. Every other provider I could run locally releases its loader after #12987; Glue is the one that does not, and its cause is different in kind, which is why it is filed on its own.

Verified against LocalStack on a packaged server started with `-Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=512m`: create a `glue` catalog, list and create a schema, drop the catalog, then force a full GC. `jcmd VM.classloader_stats` still lists the catalog's `IsolatedClassLoader$CustomURLClassLoader` with ~2265 classes, and `jcmd GC.heap_info` shows Metaspace unchanged.

Tracing the heap dump back to a GC root gives:

```
JAVA_FRAME (a live thread's stack)
-> software.amazon.awssdk.http.apache.internal.conn.IdleConnectionReaper$ReaperTask
-> --loader--> the catalog's ClassLoader
```

The thread is `idle-connection-reaper`, running `IdleConnectionReaper$ReaperTask.run`. The AWS SDK keeps that reaper as a singleton per ClassLoader and stops it only when the last connection manager is deregistered, that is, when every Apache-HTTP-backed client built by that loader has been closed. It also swallows interrupts, so the ClassLoader cleanup cannot stop it from the outside: I confirmed the cleanup does interrupt the thread and the thread keeps running.

`GlueCatalogOperations.close()` does close its `glueClient`, so the reaper staying alive says some other AWS client the catalog builds is never closed. The warehouse is an `s3a://` location, so an S3 client is the obvious candidate, but I did not chase which one.

### How should we improve?

Find the AWS client that outlives `GlueCatalogOperations.close()` and close it there, so the SDK deregisters its connection manager and the reaper stops on its own.

Please do not paper over this in `ClassLoaderResourceCleanerUtils` by reflecting into `IdleConnectionReaper` and forcing the thread down. That would hide a client that is leaking sockets and connection pools for as long as the server runs, which is a problem in its own right, and it would depend on SDK internals that are not API.

A regression test in the same shape as the manual check would be worth having: drop the catalog, force a full GC, assert that no `CustomURLClassLoader` for it remains.

Found while working on #12986 / #12987, which fix the same symptom for `hive`, `lakehouse-iceberg`, `lakehouse-paimon`, the JDBC catalogs and `fileset` on s3a, gs and abfss.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.