AbsaOSS / AbsaOSS/hyperdrive-trigger

Find out why attempt.aggs.stages.inputRecords.sum is zero, even if records were processed

Abierto
#685 1 comentario 0 reacciones 0 asignados Ver en GitHub
needs_clarification
Lenguaje dominante
TypeScript
Estrellas
8
Forks
5
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

In spark history, Find out why attempt.aggs.stages.inputRecords.sum shows zero, even if QueryProgressEvent has non-zero numInputRows, e.g. like here:

{"Event":"org.apache.spark.sql.streaming.StreamingQueryListener$QueryProgressEvent","progress":{"id":"57114e13-db5e-4dc6-930b-a7ff99c77666","runId":"c71422cb-0787-4423-bf1b-491b19e83e35","name":null,"timestamp":"2022-05-26T09:04:09.740Z","batchId":39556,"durationMs":{"triggerExecution":9331,"getEndOffset":0,"queryPlanning":254,"getBatch":8,"setOffsetRange":908,"addBatch":7738,"walCommit":195},"eventTime":{},"stateOperators":[],"sources":[{"description":"KafkaV2[Subscribe[redacted.topic]]","startOffset":"{\"redacted.topic\":{\"2\":172810,\"5\":172921,\"4\":172097,\"1\":172469,\"3\":172141,\"0\":172512}}","endOffset":"{\"redacted.topic\":{\"2\":172826,\"5\":172933,\"4\":172112,\"1\":172479,\"3\":172154,\"0\":172524}}","numInputRows":78,"inputRowsPerSecond":"NaN","processedRowsPerSecond":8.359232665309184}],"sink":

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.