[Bug] JNI scanner leaks off-heap memory and fails queries when getNext throws an unchecked exception
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
master / 4.0
### What's Wrong?
JNI scan threads are pooled and reused across queries, and a JNI exception is sticky per-thread: it stays pending until explicitly cleared. Two related problems arise:
1. FE `JniScanner.getNextBatchMeta()` only catches `IOException` from `getNext()`. Scanner subclasses (and the libraries they call) routinely surface failures as unchecked `RuntimeException`/`NullPointerException`. When such an exception escapes:
- `releaseTable()` is skipped, leaking the off-heap `VectorTable`;
- a bare pending JNI exception is left on the pooled scan thread, later misattributed to an unrelated JNI call (e.g. `getStatistics`) and able to fail an otherwise healthy query.
2. BE `JniReader::_get_statistics` / `JniReader::close` and `JniTableReader::_get_statistics` run JNI calls while a stale pending exception (left by an earlier call on the same pooled thread) may still be present — undefined behavior per the JNI spec. In `close()` the early `RETURN_ERROR_IF_EXC` aborts before the resource-release/close calls, leaking the Java scanner and the off-heap table.
### What You Expected?
An unchecked exception from `getNext()` should always release the scanner table and be normalized to a checked exception, and BE best-effort statistics/close paths should not run under a stale pending exception or leak resources.
### How to Reproduce?
Run a JNI-based scan (e.g. an external table via a JNI scanner) where `getNext()` throws an unchecked exception (a malformed record or a connector-library `RuntimeException`/NPE). The off-heap table is leaked and a subsequent unrelated JNI call on the reused scan thread can fail the query.
### Anything Else?
Fix: FE catches `Throwable`, always calls `releaseTable()`, and normalizes non-`IOException` throwables to `IOException`; BE discards any residual pending exception up front (logging it) before touching the JVM again.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with FE JniScanner.getNextBatchMeta() and the BE JniReader::_get_statistics, JniReader::close, and JniTableReader::_get_statistics paths described in the issue. Reproduce an unchecked getNext() failure through a JNI-based scan, then verify that resources are released, the throwable is normalized, and stale pending JNI exceptions do not affect later statistics or close calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100