apache / apache/kyuubi

[FEATURE] spark on k8s on s3 Running mode. Supports uploading log files to s3

Open
#6,136 1 comment 0 reactions 0 assignees View on GitHub
kind:feature priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Aws serverless spark has a feature that periodically uploads log files to s3 during task execution, facilitating log viewing and simplifying spark log collection.
Extending the open source version of spark enables similar functionality?

![image](https://github.com/apache/kyuubi/assets/1145830/f54d9529-b436-4bca-b7d7-01052da4e656)
![image](https://github.com/apache/kyuubi/assets/1145830/755bdd44-69b0-4679-a1df-e9063db8af92)

![image](https://github.com/apache/kyuubi/assets/1145830/81f6a8d5-e584-415e-877f-2647ad7e65f7)
![image](https://github.com/apache/kyuubi/assets/1145830/23726014-f6fc-4446-b1f2-dc5d1754172a)

event log:
![image](https://github.com/apache/kyuubi/assets/1145830/a9d7a456-11fd-4d31-9e27-ab1514e3c092)

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.