apache / apache/gluten

[VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode

Open
#5,320 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently, driver generate `GlutenPartition` based on spark's `FilePartitions`, and then convert to `LocalFilesNode` and serialized to byte array in pb format.
This will double the driver memory, because the `FilePartitions` are not destroyed after convert to `LocalFilesNodes`.
When there are many file splits ( file status) , the impact is significant.

For example, in one of our case, there are total 48 hdfs paths to list, 7039474 files under them. With vanilla spark, it can work with driver memory = 20G, but failed in Gluten.

From the gc log, we can find that Gluten has more `String` and `Byte[]` objects than vanilla spark.

Vanilla Spark Full GC objects
```
num #instances #bytes class name
----------------------------------------------
1: 42535479 8856286272 [C
2: 42538104 1020914496 java.lang.String
3: 7044015 563521200 java.net.URI
4: 7039474 506842128 org.apache.hadoop.fs.LocatedFileStatus
5: 13412 332304008 [B
6: 7039474 281578960 org.apache.spark.sql.execution.datasources.PartitionedFile
7: 7040016 225280512 scala.collection.mutable.LinkedHashSet$Entry
8: 7039542 225265344 scala.collection.mutable.LinkedEntry
9: 7039479 225263328 org.apache.hadoop.fs.permission.FsPermission
10: 1412 151374272 [Lscala.collection.mutable.HashEntry;
11: 145 125501688 [Lorg.apache.hadoop.fs.FileStatus;
12: 7039625 112634000 org.apache.hadoop.fs.Path
13: 55673 42854960 [Ljava.lang.Object;
14: 146968 30759312 [Lorg.apache.spark.sql.execution.datasources.PartitionedFile;
15: 2462 27069520 [J
16: 1004712 24113088 java.util.concurrent.ConcurrentSkipListMap$Node
17: 146968 16460416 org.apache.spark.scheduler.ResultTask
18: 791929 12670864 scala.Some
```
Gluten Full GC objects
```
num #instances #bytes class name
----------------------------------------------
1: 70600217 9596405088 [C
2: 153749 2117256784 [B
3: 70603033 1694472792 java.lang.String
4: 28210146 902724672 java.util.HashMap$Node
5: 7056556 564282560 [Ljava.util.HashMap$Node;
6: 7044001 563520080 java.net.URI
7: 7039474 506842128 org.apache.hadoop.fs.LocatedFileStatus
8: 7054771 338629008 java.util.HashMap
9: 7039496 225263872 scala.collection.mutable.LinkedEntry
10: 7039479 225263328 org.apache.hadoop.fs.permission.FsPermission
11: 7040463 168971112 java.lang.Long
12: 777126 135040840 [Ljava.lang.Object;
13: 7039578 112633248 org.apache.hadoop.fs.Path
14: 1332 67224064 [Lscala.collection.mutable.HashEntry;
15: 97 56405176 [Lorg.apache.hadoop.fs.FileStatus;
16: 748173 17956152 java.util.ArrayList
17: 593611 14246664 scala.collection.immutable.$colon$colon
18: 1919 9036728 [J
```

So I propose to reuse the `FilePartition` in Driver, and postpone the conversion of `LocalFilesNode` in executor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.