[VL] Delta: Slow scan when reading small files
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Issue description
On a TPC-DS dataset where 182300 files (which is a lot) generated for table `web_sales` (similarly, other fact tables all include many small files), Gluten is much slower than vanilla Spark when reading the data.
Code versions: Latest Glute code + Spark 3.4 + Delta 2.4
See test report:
```
Test report:
Summary: 5 out of 5 queries passed.
| Query ID | Passed | Row Count | Planning Time (Millis) | Query Time (Millis) | Speedup |
| | | Vanilla | Gluten | Vanilla | Gluten | Vanilla | Gluten | |
|----------|--------|---------|--------|------------|------------|-----------|-----------|---------|
| q1| true| 100| 100| 13348| 11773| 27781| 9613| 188.99%|
| q2| true| 2513| 2513| 5610| 5452| 81757| 277679| -70.56%|
| q3| true| 100| 100| 3969| 4366| 31489| 18550| 69.75%|
| q4| true| 100| 100| 2140| 2392| 300999| 658322| -54.28%|
| q5| true| 100| 100| 4013| 3674| 219881| 129141| 70.26%|
| all| true| 2913| 2913| 29080| 27657| 661907| 1093305| -39.46%|
No failed queries.
```
Test command used (gluten-it):
```
sbin/gluten-it.sh queries-compare --benchmark-type=ds --data-gen=once --local-cluster --auto-cluster-resource --off-heap-ratio=0.5 --enable-history --enable-ui --gen-partitioned-data -s=1000.0 --data-source=delta --data-dir=/root/data --extra-conf=spark.gluten.sql.columnar.scanOnly=true --queries=q1,q2,q3,q4,q5 --shuffle-partitions=100
```
Hardware (r7i.x16large + 500GiB EBS):
```
Gluten Version: 1.6.0-SNAPSHOT
Commit: a1edfafcd4025440caef8bba5a0d5a1c432c2480
CMake Version: 3.28.3
System: Linux-6.1.141-155.222.amzn2023.x86_64
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) Platinum 8488C
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 13.3.0
C Compiler: /usr/bin/cc
C Compiler Version: 13.3.0
CMake Prefix Path: /usr/local;/usr;/;/root/.local/share/uv/tools/cmake/lib/python3.12/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
```
Contributor guide
Research direction
Start by reproducing the q1–q5 comparison with the listed gluten-it command, focusing on the slower q2 and q4 scans over the Delta web_sales data. Compare planning and query times with vanilla Spark, then trace the scan path to identify the small-file bottleneck; done means the reported regression is explained and scan performance is improved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100