AbsaOSS / AbsaOSS/hyperdrive-trigger
Find out why attempt.aggs.stages.inputRecords.sum is zero, even if records were processed
- 主要語言
- TypeScript
- 星號
- 8
- 分支
- 5
- PR 合併指標
- 30 天內沒有已合併 PR
描述
In spark history, Find out why attempt.aggs.stages.inputRecords.sum shows zero, even if QueryProgressEvent has non-zero numInputRows, e.g. like here:
{"Event":"org.apache.spark.sql.streaming.StreamingQueryListener$QueryProgressEvent","progress":{"id":"57114e13-db5e-4dc6-930b-a7ff99c77666","runId":"c71422cb-0787-4423-bf1b-491b19e83e35","name":null,"timestamp":"2022-05-26T09:04:09.740Z","batchId":39556,"durationMs":{"triggerExecution":9331,"getEndOffset":0,"queryPlanning":254,"getBatch":8,"setOffsetRange":908,"addBatch":7738,"walCommit":195},"eventTime":{},"stateOperators":[],"sources":[{"description":"KafkaV2[Subscribe[redacted.topic]]","startOffset":"{\"redacted.topic\":{\"2\":172810,\"5\":172921,\"4\":172097,\"1\":172469,\"3\":172141,\"0\":172512}}","endOffset":"{\"redacted.topic\":{\"2\":172826,\"5\":172933,\"4\":172112,\"1\":172479,\"3\":172154,\"0\":172524}}","numInputRows":78,"inputRowsPerSecond":"NaN","processedRowsPerSecond":8.359232665309184}],"sink":
貢獻指南
評估
這個 Issue 還沒有評估資料。