apache / apache/pinot

Memory leak in the JDBC client

Open
#10,714 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

I'm digging into an issue with OOM errors and a memory leak of some kind using the java-client and jdbc connections to Pinot. Eventually our Spring Boot application dies with:
`Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncHttpClient-timer-1820-1"`
In digging into this I found https://github.com/AsyncHttpClient/async-http-client/issues/1658, the important part (considering I see hundreds of threads marked AsyncHttpClient-timer-X-1) of which is:
```
AsyncHttpClient-x-x: I/O (twice number of cores)
AsyncHttpClient-timer: timer for timeouts (only one)

Any different number means you're creating multiple clients.
```
I dug into the Pinot code where this issue seems to be coming from and found where it's creating an AsyncHttpClient:

- [https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]/org/apache/pinot/client/JsonAsyncHttpPinotClientTransport.java](https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/JsonAsyncHttpPinotClientTransport.java#L69)
- [https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]rg/apache/pinot/client/controller/PinotControllerTransport.java](https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/controller/PinotControllerTransport.java#L70)
- https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java#L120

The problem here as I see it is that the docs for AsyncHttpClient say
`AsyncHttpClient instances are intended to be global resources that share the same lifecycle as the application. Typically, AHC will usually underperform if you create a new client for each request, as it will create new threads and connection pools for each.`
which the Pinot java client is clearly not doing, and looking at the classes that create these client it's up to other code that's using them to close them. I think this is resulting in client (and their respective threadpools) being kept around in parallel causing memory leakage.

Contributor guide

Open the contributing guide

Research direction

Start with JsonAsyncHttpPinotClientTransport.java, PinotControllerTransport.java, and BrokerCache.java at the cited AsyncHttpClient creation points. Trace how each client and its thread pools are created and closed, then verify that repeated JDBC or Java-client use does not leave parallel clients and timer threads accumulating or reproduce the reported out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.