[FLINK] Memory leak when expression contains string constant
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
example sql
```
create table s(
a int,
b string,
c string,
d as cast(c as Timestamp(3))
) with (
'connector' = 'kafka',
'topic' = '***',
'properties.bootstrap.servers' = '****',
'properties.group.id' = 'nexmark',
'format' = 'json'
);
create table t(
d timestamp(3)
) with ('connector' = 'blackhole');
insert into t select d from s where b = 'abc';
```
the log
E0626 10:41:29.391786 3934329 Exceptions.h:66] Line: /home/zouyunhe/velox4j/src/main/cpp/build/_deps/velox-src/velox/experimental/stateful/StatefulTask.cpp:73, Function:initOperators, Expression: Unexpected memory pool allocations during stateful task[Task - EID 0] initialization:
```
Top 1 leaf memory pool usages:
op.85.0.0.FilterProject usage 128B reserved 1.00MB peak 256B
root usage 128B reserved 1.00MB peak 1.00MB
Query Memory Pool - EID 0 usage 128B reserved 1.00MB peak 1.00MB
task.Task - EID 0 usage 128B reserved 1.00MB peak 1.00MB
node.85 usage 128B reserved 1.00MB peak 1.00MB
op.85.0.0.FilterProject usage 128B reserved 1.00MB peak 256B
W0626 10:42:25.782680 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 4, Resource handle ID: 5, Description: N7velox4j7SessionE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.
W0626 10:42:25.782732 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 5, Resource handle ID: 5, Description: N7velox4j18StatefulSerialTaskE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.
I0626 10:42:25.782845 3926896 Task.cpp:2307] Terminating task Task - EID 0 with state Finished after running for 0ms
W0626 10:42:25.783104 3926896 ObjectStore.cc:34] Unclosed object [Store ID: 4, Resource handle ID: 4, Description: N7velox4j13MemoryManagerE] is found when object store is closing. Velox4J will destroy it automatically but it's recommended to manually close the object through the Java API CppObject#close() after use, to minimize peak memory pressure of the application.
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by running the SQL reproduction with the Velox backend and compare the memory-pool output. Read StatefulTask.cpp around line 73, then inspect the unclosed-object warnings from ObjectStore.cc and Task.cpp. Done means the reproduction no longer reports unexpected allocations or unclosed objects after the task finishes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100