AbsaOSS / AbsaOSS/hyperdrive-trigger
Find out why attempt.aggs.stages.inputRecords.sum is zero, even if records were processed
- 主要言語
- TypeScript
- スター
- 8
- フォーク
- 5
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
In spark history, Find out why attempt.aggs.stages.inputRecords.sum shows zero, even if QueryProgressEvent has non-zero numInputRows, e.g. like here:
{"Event":"org.apache.spark.sql.streaming.StreamingQueryListener$QueryProgressEvent","progress":{"id":"57114e13-db5e-4dc6-930b-a7ff99c77666","runId":"c71422cb-0787-4423-bf1b-491b19e83e35","name":null,"timestamp":"2022-05-26T09:04:09.740Z","batchId":39556,"durationMs":{"triggerExecution":9331,"getEndOffset":0,"queryPlanning":254,"getBatch":8,"setOffsetRange":908,"addBatch":7738,"walCommit":195},"eventTime":{},"stateOperators":[],"sources":[{"description":"KafkaV2[Subscribe[redacted.topic]]","startOffset":"{\"redacted.topic\":{\"2\":172810,\"5\":172921,\"4\":172097,\"1\":172469,\"3\":172141,\"0\":172512}}","endOffset":"{\"redacted.topic\":{\"2\":172826,\"5\":172933,\"4\":172112,\"1\":172479,\"3\":172154,\"0\":172524}}","numInputRows":78,"inputRowsPerSecond":"NaN","processedRowsPerSecond":8.359232665309184}],"sink":
コントリビューションガイド
評価
この issue はまだ評価されていません。