[Critial][Perf] dataframe cache has significantly perf regression issue when reading json file
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
We has a job which run daily to read some jsons file to do some simple join op to generate summary metrics.
The job costs increased signficantly from **10 minutes** to now **3 hours** after we sync do last sync of gluten main.

The issue only happend when call `dataframe.cache` when reading/processing velox-non supported file format as csv, json, etc.
And it can be easily reproduced with below code
```
spark.read.json(s"${jsonFilePath}/*/**").cache().collect()
```
before


after the last week sync:


The issue we found caused due to this PR [[VL] Support columnar table cache](https://github.com/oap-project/gluten/pull/3152)
the PR used for `dafaframe.cache` support but it introduces additional perf issue when cache for csv/json. In shortly, when reading those velox non-supported file, insteads of do directly convert from internelRow to cachedBatch, cachedBatch to internelRow. It always do additional C2R and R2C convert which is actually not needed .
and it can be fixed when disable `columnar table cache` feature.
## Expected behavior
For velox non-support file, please always fallback to use rowbased cachedBatch serializer - DefaultCachedBatchSerializer, insteads of `ColumnarCachedBatchSerializer` , or at least remove addtiaonal C2R and R2C convertion when reading caching for those type file to avoid additional perf issue.
### Spark version
Spark-3.3.x
### Spark configurations
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Review PR #3152 and trace the dataframe.cache path involving ColumnarCachedBatchSerializer and DefaultCachedBatchSerializer for JSON or other Velox-unsupported files. Reproduce the issue with the provided spark.read.json(...).cache().collect() example, then verify that the unnecessary C2R and R2C conversions are avoided and performance returns to the expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100