Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Load job failure related has succeed

Aberta
#33 1 comentário 0 reações 1 responsável Reivindicada por @Claudenw Ver no GitHub
Linguagem predominante
Java
Estrelas
37
Forks
45
Merge médio
19h 50min
PRs com merge (30d)
5

Descrição

I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn;t uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
1. I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
2. the file is deleted from the google storage bucket.

From my understanding and testing the issue is that job.isDone return true, since the job ended but it's not guarantee that it's ended successfully.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.

logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.