Root Allocator doesn't release memory when its closed
- Lenguaje dominante
- Java
- Estrellas
- 94
- Forks
- 152
- Merge medio
- 3 d 16 h
- PR fusionados (30 d)
- 11
Descripción
### Describe the bug, including details regarding any error messages, version, and platform.
There is a bug in the root allocator where it doesn't release the memory when it's closed. Attached the sample code that prints the RSS memory.
I ran this code with the following jvm params
```
--add-opens=java.base/java.nio=ALL-UNNAMED -Xms1g -Xmx1g -XX:+AlwaysPreTouch
```
Code
```
public final class ArrowOnlyParquetMemoryRepro {
private static final long ARROW_MEMORY_LIMIT_BYTES = 100L * 1024L * 1024L; // 100 MB
private static final int ARROW_BATCH_SIZE_ROWS = 1024 * 1024; // 1 million rows
private ArrowOnlyParquetMemoryRepro() {
}
public static void main(String[] args) throws Exception {
// Parse arguments
File parquetFile = null;
List scanCols = null;
for (int i = 0; i < args.length; i++) {
if ("--path".equals(args[i]) && i + 1 < args.length) {
parquetFile = new File(args[i + 1]);
} else if ("--scanCols".equals(args[i]) && i + 1 < args.length) {
scanCols = parseCsv(args[i + 1]);
}
}
if (parquetFile == null || scanCols == null || scanCols.isEmpty()) {
System.err.println("Usage: ArrowOnlyParquetMemoryRepro --path --scanCols ");
System.exit(1);
}
if (!parquetFile.exists() || !parquetFile.isFile()) {
System.err.println("File does not exist: " + parquetFile.getAbsolutePath());
System.exit(1);
}
log("config", "file=" + parquetFile.getAbsolutePath() + " cols=" + scanCols);
logRss("start");
// Scan each column
for (String col : scanCols) {
logRss("before_scan:" + col);
scanColumn(parquetFile, col);
logRss("after_scan:" + col);
}
logRss("end");
// Force GC and observe RSS
System.gc();
Thread.sleep(1000);
logRss("after_gc");
// Keep process alive for external memory inspection (e.g., via `ps` or `/proc//status`)
log("info", "Process will now sleep. Use 'ps -o rss,pid,command' or 'cat /proc//status' to inspect RSS.");
while (true) {
Thread.sleep(60_000);
System.gc();
logRss("sleeping");
}
}
private static void scanColumn(File parquetFile, String col) throws Exception {
long rowsScanned = 0;
try (RootAllocator allocator = new RootAllocator(ARROW_MEMORY_LIMIT_BYTES);
DatasetFactory datasetFactory = new FileSystemDatasetFactory(
allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, parquetFile.toURI().toString());
Dataset dataset = datasetFactory.finish()) {
ScanOptions options = new ScanOptions.Builder(ARROW_BATCH_SIZE_ROWS)
.columns(Optional.of(new String[]{col}))
.build();
try (Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()) {
while (reader.loadNextBatch()) {
VectorSchemaRoot root = reader.getVectorSchemaRoot();
if (root == null || root.getFieldVectors().isEmpty()) {
continue;
}
FieldVector vector = root.getVector(col);
if (vector == null) {
log("warn", "Column not found in file: " + col);
return;
}
int rowCount = root.getRowCount();
// Access values to ensure they're materialized
for (int i = 0; i < rowCount; i++) {
vector.getObject(i);
}
rowsScanned += rowCount;
}
}
}
log("scan", "col=" + col + " rowsScanned=" + rowsScanned);
}
private static List parseCsv(String csv) {
List result = new ArrayList<>();
for (String part : csv.split(",")) {
String trimmed = part.trim();
if (!trimmed.isEmpty()) {
result.add(trimmed);
}
}
return result;
}
private static void logRss(String phase) {
long rssKb = readRssKb();
if (rssKb < 0) {
log("rss", "phase=" + phase + " rssKb=N/A (not on Linux)");
} else {
log("rss", "phase=" + phase + " rssKb=" + rssKb + " rssMb=" + (rssKb / 1024));
}
}
private static long readRssKb() {
File status = new File("/proc/self/status");
if (!status.exists()) {
return -1;
}
try (BufferedReader br = new BufferedReader(new FileReader(status, StandardCharsets.UTF_8))) {
String line;
while ((line = br.readLine()) != null) {
if (line.startsWith("VmRSS:")) {
String[] parts = line.trim().split("\\s+");
if (parts.length >= 2) {
return Long.parseLong(parts[1]);
}
}
}
return -1;
} catch (Exception e) {
return -1;
}
}
private static void log(String tag, String msg) {
System.out.println("[" + tag + "] " + msg);
}
}
```
This is the output
```
[rss] phase=start rssKb=1121340 rssMb=1095
[rss] phase=before_scan:col_name rssKb=1121808 rssMb=1095
2026-01-08T10:58:01,040 INFO [main] org.apache.arrow.memory.BaseAllocator - Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
2026-01-08T10:58:01,049 INFO [main] org.apache.arrow.memory.DefaultAllocationManagerOption - allocation manager type not specified, using netty as the default type
2026-01-08T10:58:01,076 INFO [main] org.apache.arrow.memory.CheckAllocator - Using DefaultAllocationManager at
[scan] col=col_name rowsScanned=2734751
[rss] phase=after_scan:col_name rssKb=1313364 rssMb=1282
[rss] phase=before_scan:col_name rssKb=1313364 rssMb=1282
[scan] col=col_name rowsScanned=2734751
[rss] phase=after_scan:col_name rssKb=1331144 rssMb=1299
[rss] phase=before_scan:col_name rssKb=1331144 rssMb=1299
[scan] col=col_name rowsScanned=2734751
[rss] phase=after_scan:col_name rssKb=1338228 rssMb=1306
[rss] phase=end rssKb=1338228 rssMb=1306
[rss] phase=after_gc rssKb=1338228 rssMb=1306
[info] Process will now sleep. Use 'ps -o rss,pid,command' or 'cat /proc//status' to inspect RSS.
[rss] phase=sleeping rssKb=1338228 rssMb=1306
```
Guía de contribución
Línea de trabajo
Empieza con el ArrowOnlyParquetMemoryRepro proporcionado y el método scanColumn y, a continuación, reproduce el crecimiento de RSS mientras cierras cada RootAllocator. Rastrea la propiedad del allocator a través de FileSystemDatasetFactory, Dataset, Scanner y ArrowReader. La tarea estará completada cuando los escaneos repetidos liberen la memoria asignada después de que se cierren los ámbitos try-with-resources y la RSS observada deje de crecer de forma inesperada.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- backend, performance
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100