[Bug] kyuubi.operation.result.saveToFile and orc.compression leads to CPU overload on Kyuubi Server
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
When kyuubi.operation.result.saveToFile is enabled, the result is written to HDFS in ZSTD ORC format. This causes a high CPU load on the Kyuubi Server during decompression, reducing performance by hundreds of times. Additionally, Kyuubi ignores the spark.sql.orc.compression.codec=none setting.
### Affects Version(s)
1.9.2
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by tracing how kyuubi.operation.result.saveToFile writes results to HDFS in ZSTD ORC format, then inspect how spark.sql.orc.compression.codec=none is handled. Reproduce the behavior on Kyuubi 1.9.2 and compare Kyuubi Server CPU usage and output compression settings; done means avoiding the excessive decompression load and honoring the configured codec.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- backend, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100