AbsaOSS / AbsaOSS/hyperdrive-trigger

Support for Spark 3

未關閉
#719 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
TypeScript
星號
8
分支
5
PR 合併指標
30 天內沒有已合併 PR

描述

Add support for Spark 3

Needed because of #717. Spark 3 jobs cannot be submitted with spark-submit 2 and vice versa.

- Support for Spark 2 still needed
- Two artefacts should be created and deployed, one for spark 2, one for spark 3
- If all used APIs are compatible, then just add a profile so hyperdrive-trigger can be built against spark 2, and spark 3 resp.
- Otherwise, two branches can be maintained
-

# Preparation work
Use minimal app spark launcher (not full blown hyperdrive-trigger) to test spark 3 compatibility (with EMR)

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。