apache / apache/carbondata

JNI local reference leak in CarbonWriter (sortBy can overflow local ref table)

Open
#4,402 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
694
PR merge metrics
No merged PRs in 30d

Description

## Problem

`sdk/CSDK/src/CarbonWriter.cpp` creates JNI local references (`NewStringUTF`, `FindClass`, `NewObjectArray`, `GetObjectClass`, `CallObjectMethodA`) and almost never calls `DeleteLocalRef`.

The C++ SDK embeds a JVM and keeps `JNIEnv` for the whole writer lifetime. These references are **not** released when a C++ function returns. They stay in the local reference table until `DeleteLocalRef`, `PopLocalFrame`, thread detach, or JVM destroy.

`close()` only deletes `carbonWriterBuilderObject`, `carbonWriterObject`, and a *local* `jclass` that shadows the member. Temporary refs from builder APIs are never cleaned up.

## Impact

- `sortBy` allocates one `jstring` per column in a loop and never deletes it after `SetObjectArrayElement`. A large `argc` can overflow the JNI local reference table (`JNI local references: ... exceeded`).
- Other builder methods (`outputPath`, `withHadoopConf`, `withLoadOption`, `withTableProperty`, `writtenBy`, …) leak a few refs per call. They persist until JVM destroy.
- Member fields store local refs without `NewGlobalRef`, which is invalid JNI for objects that must outlive a single native frame.

## Affected code

[`sdk/CSDK/src/CarbonWriter.cpp`](https://github.com/apache/carbondata/blob/f86ac085ddbbd8381b0a5b65658731fe618eff4a/sdk/CSDK/src/CarbonWriter.cpp)

**`sortBy` — leak grows with column count:**

```cpp
jclass objectArrayClass = jniEnv->FindClass("Ljava/lang/String;");
jobjectArray array = jniEnv->NewObjectArray(argc, objectArrayClass, NULL);
for (int i = 0; i < argc; ++i) {
jstring value = jniEnv->NewStringUTF(argv[i]);
jniEnv->SetObjectArrayElement(array, i, value);
// missing: jniEnv->DeleteLocalRef(value);
}
carbonWriterBuilderObject = jniEnv->CallObjectMethodA(carbonWriterBuilderObject, methodID, args);
// missing: DeleteLocalRef(objectArrayClass / array / previous builder / carbonWriterBuilderClass)

Same pattern in outputPath (NewStringUTF + overwrite builder object) and withHadoopConf / withTableProperty / withLoadOption (two NewStringUTF args, never deleted).

close() does not cover these temps, and the member class is not deleted:
jclass carbonWriter = jniEnv->GetObjectClass(carbonWriterObject); // shadows the member
// ...
jniEnv->DeleteLocalRef(carbonWriterBuilderObject);
jniEnv->DeleteLocalRef(carbonWriterObject);
jniEnv->DeleteLocalRef(carbonWriter); // deletes the local jclass, not the FindClass member

Related: FindClass("Ljava/lang/String;") is the wrong name; JNI class names use java/lang/String.

Suggested fix
1. After each use, DeleteLocalRef on temps (jstring, jclass from GetObjectClass/FindClass, jobjectArray). In sortBy, delete value every iteration; delete array / objectArrayClass after the Java call.
2. Before overwriting carbonWriterBuilderObject, delete the previous local ref (or keep one global ref and update it).
3. Promote long-lived members (carbonWriter, carbonWriterBuilderObject, carbonWriterObject) with NewGlobalRef, and DeleteGlobalRef in close().
4. Prefer PushLocalFrame / PopLocalFrame or a small RAII local-ref wrapper so exception / early-return paths cannot skip cleanup.
5. Change FindClass("Ljava/lang/String;") to FindClass("java/lang/String").

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in sdk/CSDK/src/CarbonWriter.cpp, focusing on sortBy and the builder methods named in the issue, then inspect close() and the member reference handling. Trace each temporary and long-lived JNI reference through normal and cleanup paths. Done means temporary references are released, persistent members use the appropriate lifetime, the String class lookup is corrected, and sortBy no longer grows the local reference table with each column.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.