duckdb / duckdb/duckdb-node

Memory leak while reading parquet files from disk and web

Open
#22 12 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
91
Forks
36
PR merge metrics
No merged PRs in 30d

Description

**What happens?**
I have a service that operates reading parquet files from disk and web (over HTTP). I believe that there is a memory leak causing node.js to run out of memory.

**To Reproduce**
The following code read a small parquet file, only 1 line of data, from disk repeatedly and logs the memory usage. We can observe a increase of both rss and heap over time. I also created a graph containing the first 54000 iterations. Reading bigger parquet files leads to a bigger leak from what I have seen.

```
const duckdb = require('duckdb');

const db = new duckdb.Database(':memory:');

let i = 0;
function test() {
db.all("SELECT * FROM READ_PARQUET('file.parquet')", () => {
i += 1;
const memory = process.memoryUsage();
console.log('Iteration: ', i, memory);

test()
});
}

test()
```

![image](https://github.com/duckdb/duckdb-node/assets/36680772/1fb182c9-511f-4070-8162-69f46aa023d5)

**OS:**
Both windows and ubuntu

**DuckDB Version:**
0.9.1

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.