Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Load job failure related has succeed
- Linguagem predominante
- Java
- Estrelas
- 37
- Forks
- 45
- Merge médio
- 19h 50min
- PRs com merge (30d)
- 5
Descrição
I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn;t uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
1. I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
2. the file is deleted from the google storage bucket.
From my understanding and testing the issue is that job.isDone return true, since the job ended but it's not guarantee that it's ended successfully.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.