apache / apache/datafusion

Potential memory issue when using COPY with PARTITIONED BY

Open
#11,042 18 comments 3 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Memory does not get freed after executing multiple `COPY ... TO ... PARTITIONED BY ...` queries. I have not been able to identify what is causing this behavior.

### To Reproduce

The behavior can be observed using datafusion-cli. I have been monitoring the memory usage through Activity Monitor.

1. Download test parquet file (120MB): https://file.io/eKiHwu4waHVN
2. Run `datafusion-cli`
3. Create a external table:

```
CREATE EXTERNAL TABLE my_table
(
col1 VARCHAR NOT NULL,
timestamp BIGINT NOT NULL,
col2 VARCHAR NOT NULL,
col3 VARCHAR NOT NULL,
col4 VARCHAR NOT NULL,
col5 VARCHAR NOT NULL,
col6 VARCHAR NOT NULL,
col7 VARCHAR NOT NULL,
col8 VARCHAR NOT NULL,
col9 VARCHAR NOT NULL,
col10 VARCHAR NOT NULL,
col11 VARCHAR NOT NULL,
col12 DOUBLE
)
WITH ORDER (col1 ASC, timestamp ASC) STORED AS PARQUET LOCATION 'test_file.parquet';
```

4. Execute `COPY .. PARTITIONED BY` query:

```
COPY (SELECT col1, timestamp, col10, col12 FROM my_table ORDER BY col1 ASC, timestamp ASC)
TO './output' STORED AS PARQUET PARTITIONED BY (col1) OPTIONS (compression 'uncompressed');
```

5. Monitor memory usage.
6. Repeat execution of `COPY .. PARTITIONED BY` query and continue monitoring memory usage.
7. Observation: memory does not get released.

### Expected behavior

My expectation is to be able to run the `COPY` command multiple times without having the memory usage increasing every time.

### Additional context

There is more context of what I am trying to do in Discord: https://discord.com/channels/885562378132000778/1166447479609376850/1253419900043526236

I am also experiencing the same behavior when running my application in Kubernetes. K8s terminates my pod once it exceeds the pod memory limits:

Screenshot 2024-06-20 at 8 46 20 PM

Contributor guide

Open the contributing guide

Research direction

Start with datafusion-cli and reproduce the issue using the external table and repeated COPY ... TO ... PARTITIONED BY queries described here. Monitor memory across repeated executions and compare the result with the expected behavior: memory usage should not increase each time.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.