apache / apache/kyuubi

[Improvement] Remove SparkConsoleProgressBar in favor of SparkProgressMonitor

Open
#2,932 1 comment 0 reactions 1 assignee Claimed by @wForget View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

Now in spark engine, we have two implementations of progress,
In #2119, ported the implementation of Spark ConsoleProgressBar, using `KyuubiStatement#getQueryLog` to pull logs.
In #2493, ported some implementations of Hive on Spark, using `GetOperationStatus`.

The functionality is duplicated, we can remove the implementation of `SparkConsoleProgressBar`.

### How should we improve?

https://github.com/apache/incubator-kyuubi/pull/2923#issuecomment-1161337819

- Introduce configuration to decide whether progress is output to `TProgressUpdateResp` or written to operation log, which is compatible with more hive client versions.
- If a query generates many stages, using `SparkProgressMonitor` will output the progress of each stage to the operation log, and the log will be larger.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.