apache / apache/kyuubi

[Bug] kyuubi.operation.result.saveToFile and orc.compression leads to CPU overload on Kyuubi Server

Open
#7,115 2 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

When kyuubi.operation.result.saveToFile is enabled, the result is written to HDFS in ZSTD ORC format. This causes a high CPU load on the Kyuubi Server during decompression, reducing performance by hundreds of times. Additionally, Kyuubi ignores the spark.sql.orc.compression.codec=none setting.

### Affects Version(s)

1.9.2

### Kyuubi Server Log Output

```logtalk

```

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how kyuubi.operation.result.saveToFile writes results to HDFS in ZSTD ORC format, then inspect how spark.sql.orc.compression.codec=none is handled. Reproduce the behavior on Kyuubi 1.9.2 and compare Kyuubi Server CPU usage and output compression settings; done means avoiding the excessive decompression load and honoring the configured codec.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark
Domain
backend, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.