apache / apache/gluten

[FLINK] Memory leak when expression contains string constant

Open
#10,060 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

example sql
```
create table s(
a int,
b string,
c string,
d as cast(c as Timestamp(3))
) with (
'connector' = 'kafka',
'topic' = '***',
'properties.bootstrap.servers' = '****',
'properties.group.id' = 'nexmark',
'format' = 'json'
);

create table t(
d timestamp(3)
) with ('connector' = 'blackhole');

insert into t select d from s where b = 'abc';
```

the log
E0626 10:41:29.391786 3934329 Exceptions.h:66] Line: /home/zouyunhe/velox4j/src/main/cpp/build/_deps/velox-src/velox/experimental/stateful/StatefulTask.cpp:73, Function:initOperators, Expression: Unexpected memory pool allocations during stateful task[Task - EID 0] initialization:
```
Top 1 leaf memory pool usages:
op.85.0.0.FilterProject usage 128B reserved 1.00MB peak 256B

root usage 128B reserved 1.00MB peak 1.00MB
Query Memory Pool - EID 0 usage 128B reserved 1.00MB peak 1.00MB
task.Task - EID 0 usage 128B reserved 1.00MB peak 1.00MB
node.85 usage 128B reserved 1.00MB peak 1.00MB
op.85.0.0.FilterProject usage 128B reserved 1.00MB peak 256B

W0626 10:42:25.782680 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 4, Resource handle ID: 5, Description: N7velox4j7SessionE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.
W0626 10:42:25.782732 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 5, Resource handle ID: 5, Description: N7velox4j18StatefulSerialTaskE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.
I0626 10:42:25.782845 3926896 Task.cpp:2307] Terminating task Task - EID 0 with state Finished after running for 0ms
W0626 10:42:25.783104 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 4, Resource handle ID: 4, Description: N7velox4j13MemoryManagerE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.

```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by running the SQL reproduction with the Velox backend and compare the memory-pool output. Read StatefulTask.cpp around line 73, then inspect the unclosed-object warnings from ObjectStore.cc and Task.cpp. Done means the reproduction no longer reports unexpected allocations or unclosed objects after the task finishes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.