AbsaOSS / AbsaOSS/spline

Publishing Non spark Automic/Airflow Job information in spline

Đang mở
#1,150 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
investigating
Ngôn ngữ chính
Scala
Star
667
Fork
155
Merge trung bình
3 ngày 6 giờ
Pull request đã merge (30 ngày)
4

Mô tả

## Background [Optional]

We have requirement to capture the job name ( Airflow job or Automic job or any scheduler ) which is submitted . Right now we capture the spark application name which shows in the `progress` collection against `applicationName` key.
Every spark job would have submitted by scheduler ( airflow or Automic). We would like to captures this information as the top level source and the show the spark `applicationName`. I do see `extra` key in the `progress` document. but I don't see any reserved field to be displayed in the UI. It's not only Automic/airflow job name . In future We would like to also add some additional information when we progress further.
Please give some insights what's the right way showing this non spark information. Also it using the same job name we would like to search in UI.
Reason for this requirement to identify the spark job is submitted from which scheduler job. By adding this feature more users can adopt

## Question

Please give some insights what's the right way showing non spark job level information and how to make that job is searchable

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.