AbsaOSS / AbsaOSS/hyperdrive-trigger
Find out why attempt.aggs.stages.inputRecords.sum is zero, even if records were processed
- 主要语言
- TypeScript
- 星标
- 8
- 派生
- 5
- PR 合并指标
- 30 天内没有已合并 PR
描述
In spark history, Find out why attempt.aggs.stages.inputRecords.sum shows zero, even if QueryProgressEvent has non-zero numInputRows, e.g. like here:
{"Event":"org.apache.spark.sql.streaming.StreamingQueryListener$QueryProgressEvent","progress":{"id":"57114e13-db5e-4dc6-930b-a7ff99c77666","runId":"c71422cb-0787-4423-bf1b-491b19e83e35","name":null,"timestamp":"2022-05-26T09:04:09.740Z","batchId":39556,"durationMs":{"triggerExecution":9331,"getEndOffset":0,"queryPlanning":254,"getBatch":8,"setOffsetRange":908,"addBatch":7738,"walCommit":195},"eventTime":{},"stateOperators":[],"sources":[{"description":"KafkaV2[Subscribe[redacted.topic]]","startOffset":"{\"redacted.topic\":{\"2\":172810,\"5\":172921,\"4\":172097,\"1\":172469,\"3\":172141,\"0\":172512}}","endOffset":"{\"redacted.topic\":{\"2\":172826,\"5\":172933,\"4\":172112,\"1\":172479,\"3\":172154,\"0\":172524}}","numInputRows":78,"inputRowsPerSecond":"NaN","processedRowsPerSecond":8.359232665309184}],"sink":
贡献指南
评估
这个 Issue 还没有评估数据。