Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Load job failure related has succeed

Aperta
#33 1 commento 0 reazioni 1 assegnatario Rivendicata da @Claudenw Vedi su GitHub
Lingua principale
Java
Stelle
37
Fork
45
Merge medio
19h 50m
PR unite (30g)
5

Descrizione

I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn;t uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
1. I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
2. the file is deleted from the google storage bucket.

From my understanding and testing the issue is that job.isDone return true, since the job ended but it's not guarantee that it's ended successfully.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.

logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.