apache / apache/gluten

[CH] UT failed due to exceeding memory, but is ok in the prevois

Open
#7,726 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (CH)

### Bug description

Run test SQL in `GlutenClickHouseTPCHColumnarShuffleParquetAQESuite`
```sql
select LINEITEM.L_DISCOUNT,
PART.P_TYPE,
LINEITEM.L_COMMENT,
LINEITEM.L_SUPPKEY,
PART.P_PARTKEY,
PART.P_SIZE,
LINEITEM.L_RETURNFLAG,
LINEITEM.L_RECEIPTDATE,
PART.P_NAME,
SUPPLIER.S_COMMENT,
LINEITEM.L_ORDERKEY,
PART.P_MFGR,
SUPPLIER.S_ACCTBAL,
SUPPLIER.S_SUPPKEY,
LINEITEM.L_SHIPMODE,
SUPPLIER.S_NATIONKEY,
LINEITEM.L_SHIPDATE,
LINEITEM.L_COMMITDATE,
SUPPLIER.S_NAME,
PART.P_COMMENT,
LINEITEM.L_TAX,
LINEITEM.L_QUANTITY,
LINEITEM.L_PARTKEY,
PART.P_CONTAINER,
MIN(LINEITEM.L_EXTENDEDPRICE),
COUNT(LINEITEM.L_QUANTITY),
COUNT(DISTINCT LINEITEM.L_PARTKEY),
MIN(LINEITEM.L_TAX),
MIN(ORDERS.O_TOTALPRICE),
COUNT(LINEITEM.L_EXTENDEDPRICE),
COUNT(ORDERS.O_SHIPPRIORITY),
COUNT(1),
MAX(LINEITEM.L_DISCOUNT)
from LINEITEM
INNER JOIN SUPPLIER AS SUPPLIER ON LINEITEM.L_SUPPKEY = SUPPLIER.S_SUPPKEY
INNER JOIN PART AS PART ON LINEITEM.L_PARTKEY = PART.P_PARTKEY
INNER JOIN ORDERS AS ORDERS ON LINEITEM.L_ORDERKEY = ORDERS.O_ORDERKEY
where (not (((P_RETAILPRICE is not null or
((S_NATIONKEY is not null and P_MFGR like '%Manufacturer#1') or P_BRAND not like 'Brand#11')) or
((S_SUPPKEY not in
(1206, 1496, 1191, 2445, 491, 1407, 1969, 261, 1418, 310, 2099, 1343, 327, 261, 707, 37, 753, 696, 1363,
628, 1158, 2239, 26, 1180, 2448, 1698, 2099, 1326, 1247, 1203, 161, 1698, 310, 692, 491, 1920, 28, 370,
370, 261, 2258, 1146, 983, 683, 24, 1611, 5161, 3141, 2258, 1287, 683, 1720, 1887, 310, 707, 1836, 1287,
2065, 1859, 1203, 1611, 1835, 2099, 701, 2314, 692, 1418, 2367, 425, 1720, 8285, 1969, 1804, 310, 2258,
1418, 463, 2048, 368, 1253, 549, 2258, 327, 1973, 817) and 1300 > S_SUPPKEY) or (S_PHONE not in
('10-246-381-9259',
'10-211-466-9198',
'10-509-209-3829',
'10-741-929-4244',
'10-393-500-3856',
'10-495-104-1252',
'10-983-665-2259',
'10-295-590-8708',
'10-983-665-2259',
'10-745-572-7198',
'10-384-209-1825',
'10-734-420-5738',
'10-845-970-4551',
'10-630-928-4130',
'10-325-193-7475',
'%10-475-868-5521',
'10-903-990-3612',
'10-352-443-2162%',
'10-842-403-7954',
'10-789-325-3069',
'10-996-906-4890',
'10-404-519-2270',
'10-848-716-8078',
'10-246-381-9259',
'10-262-377-2302',
'10-361-729-1693',
'10-745-572-7198',
'10-384-209-1825',
'10-262-132-6639',
'10-361-729-1693',
'10-746-144-5600',
'10-409-763-8909',
'10-123-465-1292',
'10-745-572-7198%',
'10-599-740-9848',
'10-453-843-1585',
'10-191-563-6127',
'10-848-716-8078',
'10-763-945-1271',
'10-393-500-3856') and
(not (P_NAME not like 'light dark lemon lace medium%' and P_NAME is null))))) or
((((S_ADDRESS is null or P_CONTAINER in
('LG JAR', 'JUMBO CASE', 'JUMBO CASE', 'MED BOX', 'WRAP BAG', 'SM CASE',
'WRAP JAR', 'JUMBO PKG', 'SM CAN', 'SM BOX', 'JUMBO CASE', 'MED BOX', 'LG JAR',
'JUMBO CASE', 'MED DRUM', 'JUMBO PKG', 'SM CAN', 'WRAP JAR', 'LG CASE', 'LG BAG',
'SM PACK', 'JUMBO DRUM', 'WRAP BOX', 'JUMBO CAN', 'LG PKG', 'WRAP CAN',
'MED PACK', 'SM BOX', 'SM DRUM', 'SM PACK', 'MED DRUM', 'MED PACK', 'MED BOX',
'MED CAN%', 'SM JAR', 'SM CAN', 'JUMBO BOX', 'JUMBO BAG', 'LG BAG', 'LG PKG',
'LG PACK', 'LG BAG', 'JUMBO BOX', 'SM BOX', 'JUMBO CAN', 'JUMBO PKG', 'LG BAG',
'MED BOX', 'JUMBO CASE', 'MED BOX', 'LG BAG', 'LG PACK', 'MED BOX', 'LG PKG',
'SM BOX', 'WRAP BOX', 'LG CASE', 'MED PACK', 'LG PKG', '%LG CASE', 'LG JAR',
'LG BAG', 'LG BOX', 'SM CAN', 'WRAP CAN', 'WRAP PACK', 'JUMBO CASE', 'SM BOX',
'SM PACK', 'WRAP PKG', 'MED CAN', 'SM BOX', 'LG CASE', 'JUMBO CAN', 'LG JAR',
'SM DRUM', 'MED PKG', 'JUMBO BAG', 'SM CASE', 'MED BAG', 'SM PACK',
'SM PACK')) and S_SUPPKEY is not null) and (P_PARTKEY not in
(1358682, 1592117, 1114403, 839396,
1114617, 959268, 1114713, 1358631,
806397, 959018, 1114926, 812800,
1568237, 959088, 839340, 959419,
1115053, 1358740, 1114282) and
(S_SUPPKEY between 463 and 1887 or
S_SUPPKEY not in
(1287, 1422, 1878, 1191, 1804, 476,
1097, 1326, 1597, 1158, 261, 1689,
1493, 2314, 817, 1097, 2239, 327,
1887, 118, 1547, 476, 2131, 1247,
1496, 1698, 1717, 454, 1692, 1920,
1973, 2010, 1804, 774, 1611, 425,
28, 1611, 183, 983, 800, 5915, 1311,
24, 2298, 118, 183, 784, 1592, 1549,
983, 1283, 1418, 291, 118, 1407,
2072, 291, 1180, 1404, 1097, 1724,
1611, 692, 491, 316, 161, 2314,
1404, 696, 2072, 2072, 491, 1692,
764, 742, 118, 425)))) and
(P_CONTAINER in
('SM PKG', 'LG PKG', 'LG CASE', 'MED PKG', 'WRAP JAR', 'LG BAG', 'SM BOX', 'JUMBO BOX', 'SM PKG',
'SM PKG', 'JUMBO BOX', 'MED BOX', 'JUMBO PKG', 'WRAP CAN', 'MED DRUM', 'MED JAR', 'SM BAG', 'MED CAN',
'SM PACK', 'SM CASE', 'MED BAG', 'JUMBO PKG', 'LG CASE', 'SM PKG', 'MED BOX', 'LG CASE', 'JUMBO DRUM',
'MED BAG', 'JUMBO CASE', 'SM BOX', 'JUMBO PACK', 'WRAP BOX', '%JUMBO BOX', 'JUMBO BOX', 'JUMBO CASE',
'SM CAN', 'JUMBO BOX', 'SM CAN', 'LG CASE') and P_BRAND is null))))
or (((P_BRAND is not null or (P_SIZE not in
(25, 11, 48, 15, 48, 16, 3, 45, 37, 42, 47, 42, 42, 16, 97, 16, 48, 12, 87, 13, 27, 22,
42, 37, 50, 9, 34) and S_NATIONKEY >= 0)) or 955.65 = S_ACCTBAL) or
(P_TYPE not like 'MEDIUM POLISHED STEEL%' or (not S_ACCTBAL is null)))
group by LINEITEM.L_DISCOUNT, PART.P_TYPE, LINEITEM.L_COMMENT, LINEITEM.L_SUPPKEY, PART.P_PARTKEY, PART.P_SIZE,
LINEITEM.L_RETURNFLAG, LINEITEM.L_RECEIPTDATE, PART.P_NAME, SUPPLIER.S_COMMENT, LINEITEM.L_ORDERKEY,
PART.P_MFGR, SUPPLIER.S_ACCTBAL, SUPPLIER.S_SUPPKEY, LINEITEM.L_SHIPMODE, SUPPLIER.S_NATIONKEY,
LINEITEM.L_SHIPDATE, LINEITEM.L_COMMITDATE, SUPPLIER.S_NAME, PART.P_COMMENT, LINEITEM.L_TAX,
LINEITEM.L_QUANTITY, LINEITEM.L_PARTKEY, PART.P_CONTAINER
```
Get the following error, it is ok in the previous codes!

```
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Memory limit exceeded: would use 512.29 MiB (attempt to allocate chunk of 4224948 bytes), current RSS 1.21 GiB, maximum: 512.00 MiB.: While executing GraceAggregatingTransform
0. ..../SourceCode/rebase_ch/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000014561359
1. ..../SourceCode/rebase_ch/src/Common/Exception.cpp:109: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c2aa3d9
2. ..../SourceCode/rebase_ch/src/Common/Exception.h:110: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000006790a2c
3. ..../SourceCode/rebase_ch/src/Common/Exception.h:128: DB::Exception::Exception>>(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity>>::type>, char const*&&, char const*&&, String&&, long&, String&&, String&&, char const*&&, std::basic_string_view>&&) @ 0x000000000c2d10c9
4. ..../SourceCode/rebase_ch/src/Common/MemoryTracker.cpp:316: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c2d00d2
5. ..../SourceCode/rebase_ch/src/Common/MemoryTracker.cpp:373: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000c2cfc96
6. ..../SourceCode/rebase_ch/src/Common/CurrentMemoryTracker.cpp:64: CurrentMemoryTracker::allocImpl(long, bool) @ 0x000000000c27899e
7. ..../SourceCode/rebase_ch/src/Common/Allocator.cpp:233: Allocator::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x000000000c2772db
8. void DB::PODArrayBase<8ul, 4096ul, Allocator, 63ul, 64ul>::realloc<>(unsigned long) @ 0x0000000006884976
9. DB::ColumnString::insertFrom(DB::IColumn const&, unsigned long) @ 0x000000000f01fecd
10. ..../SourceCode/rebase_ch/src/Columns/ColumnNullable.cpp:273: DB::ColumnNullable::insertFrom(DB::IColumn const&, unsigned long) @ 0x0000000010c1bd1f
11. ..../SourceCode/rebase_ch/src/Columns/IColumn.cpp:149: DB::IColumnHelper::scatter(unsigned long, DB::PODArray, 63ul, 64ul> const&) const @ 0x0000000010e80b79
12. ..../SourceCode/rebase_ch/src/Interpreters/JoinUtils.cpp:569: DB::JoinCommon::scatterBlockByHash(std::vector> const&, DB::Block const&, unsigned long) @ 0x0000000010369286
13. ..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:268: local_engine::GraceAggregatingTransform::scatterBlock(DB::Block const&) @ 0x000000000c805f63
14. ..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:495: local_engine::GraceAggregatingTransform::mergeOneBlock(DB::Block const&, bool) @ 0x000000000c80285f
15. ..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:383: local_engine::GraceAggregatingTransform::prepareBucketOutputBlocks(unsigned long) @ 0x000000000c803453
16. ..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:158: local_engine::GraceAggregatingTransform::work() @ 0x000000000c8021c8
17. ..../SourceCode/rebase_ch/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::ExecutionThreadContext::executeTask() @ 0x00000000118b16a2
18. ..../SourceCode/rebase_ch/src/Processors/Executors/PipelineExecutor.cpp:289: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000118a695f
19. ..../SourceCode/rebase_ch/src/Processors/Executors/PipelineExecutor.cpp:163: DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000118a63c9
20. ..../SourceCode/rebase_ch/src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x00000000118b8134
21. ..../SourceCode/rebase_ch/src/Processors/Executors/PullingPipelineExecutor.cpp:65: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x00000000118b8299
22. ..../SourceCode/rebase_ch/utils/extern-local-engine/Parser/LocalExecutor.cpp:68: local_engine::LocalExecutor::hasNext() @ 0x000000000c672db1
23. ..../SourceCode/rebase_ch/utils/extern-local-engine/local_engine_jni.cpp:308: Java_org_apache_gluten_vectorized_BatchIterator_nativeHasNext @ 0x0000000006776237

at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
at org.apache.gluten.backendsapi.clickhouse.CollectMetricIterator.hasNext(CHIteratorApi.scala:349)
at org.apache.gluten.vectorized.CloseableCHColumnBatchIterator.$anonfun$hasNext$1(CloseableCHColumnBatchIterator.scala:42)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at org.apache.gluten.metrics.GlutenTimeMetric$.withNanoTime(GlutenTimeMetric.scala:41)
at org.apache.gluten.vectorized.CloseableCHColumnBatchIterator.hasNext(CloseableCHColumnBatchIterator.scala:42)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1931)
at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1274)
at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1274)
at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2268)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.