apache / apache/gluten

[Critial][Perf] dataframe cache has significantly perf regression issue when reading json file

Open
#3,456 10 comments 2 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

We has a job which run daily to read some jsons file to do some simple join op to generate summary metrics.
The job costs increased signficantly from **10 minutes** to now **3 hours** after we sync do last sync of gluten main.

![image](https://github.com/oap-project/gluten/assets/9858245/f9ff2e7e-d7f7-4494-9e5c-9011b46ec526)

The issue only happend when call `dataframe.cache` when reading/processing velox-non supported file format as csv, json, etc.
And it can be easily reproduced with below code

```
spark.read.json(s"${jsonFilePath}/*/**").cache().collect()
```

before
![image](https://github.com/oap-project/gluten/assets/9858245/19bf6035-71b5-4b57-a97b-26cddd667797)
![image](https://github.com/oap-project/gluten/assets/9858245/84ce7a57-9362-4d90-8fcd-8e4226ab257b)

after the last week sync:

![image](https://github.com/oap-project/gluten/assets/9858245/f414ef84-9159-482d-8745-000e32251e10)
![image](https://github.com/oap-project/gluten/assets/9858245/ff2b2dcc-ae45-4cb0-80ea-ed93275aabb9)

The issue we found caused due to this PR [[VL] Support columnar table cache](https://github.com/oap-project/gluten/pull/3152)
the PR used for `dafaframe.cache` support but it introduces additional perf issue when cache for csv/json. In shortly, when reading those velox non-supported file, insteads of do directly convert from internelRow to cachedBatch, cachedBatch to internelRow. It always do additional C2R and R2C convert which is actually not needed .

and it can be fixed when disable `columnar table cache` feature.

## Expected behavior

For velox non-support file, please always fallback to use rowbased cachedBatch serializer - DefaultCachedBatchSerializer, insteads of `ColumnarCachedBatchSerializer` , or at least remove addtiaonal C2R and R2C convertion when reading caching for those type file to avoid additional perf issue.

### Spark version

Spark-3.3.x

### Spark configurations

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Review PR #3152 and trace the dataframe.cache path involving ColumnarCachedBatchSerializer and DefaultCachedBatchSerializer for JSON or other Velox-unsupported files. Reproduce the issue with the provided spark.read.json(...).cache().collect() example, then verify that the unnecessary C2R and R2C conversions are avoided and performance returns to the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.