airbytehq / airbytehq/airbyte

SourceReader can hang after destination failure due to blocking reads ignoring cancellation

Abierto
#71,769 5 comentarios 0 reacciones 0 asignados Ver en GitHub
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Helm Chart Version

2.0.19

### What step the error happened?

During the Sync

### Relevant information

## Summary
On retries where the destination fails (e.g., BigQuery 500/400), the replication tasks usually abort quickly: the destination reader exits, the source reader gets a broken pipe and stops, and the attempt retries. However, in a subset of attempts, the destination fails but the source reader keeps running even though the MessageProcessor closes the source and destination queues. The source continues reading and emitting from the socket for hours until the pod is manually terminated.

The most likely cause is that `SourceReader` (and `DestinationReader`) block inside `attemptRead()`/`isFinished()` (backed by `Iterator.hasNext()`), and those calls are not interruptible, so coroutine cancellation does not abort the read. This leaves the source container running and the reader loop stuck.

## Environment
- Platform: airbyte-platform-v2
- Orchestrator mode: container orchestrator
- Destination: BigQuery
- Source: JDBC (bulk)

## Steps to Reproduce
1. Run a replication that produces many records from a JDBC source into BigQuery.
2. Cause the destination to fail (e.g., BigQuery 500 or 400 error during load/flush).
3. Observe that on some attempts, destination failure causes an immediate source broken pipe and fast retry.
4. On a specific attempt, observe the destination failure, MessageProcessor closes queues, but SourceReader keeps reading/printing “Read N messages” indefinitely.

## Expected Behavior
- Destination failure triggers cancellation and both DestinationReader and SourceReader stop promptly.
- Source container is terminated quickly, and the attempt retries without hanging.

## Actual Behavior
- Destination fails, MessageProcessor closes queues and finishes.
- SourceReader keeps running, continues reading from the socket, and does not terminate.
- job-pod has to be manually terminated.

## Suspected Root Cause
`SourceReader` and `DestinationReader` call `attemptRead()` and `isFinished()` in a tight loop. Those calls are backed by `Iterator.hasNext()` which can block on socket reads. Because they are not interruptible, coroutine cancellation does not stop the reader when the destination fails and the queue is closed. This can leave the source reader stuck in a blocking read and the source container running.

## Proposed Fix
Wrap blocking reads in `runInterruptible` (or equivalent) and avoid calling `isFinished()` in the loop condition, checking it only after an empty read. This allows coroutine cancellation to interrupt blocking I/O and ensures the reader loop exits promptly.

## Impact
- Attempts can hang for hours
- Unnecessary compute and resource usage
- Sync is blocked until the pod is manually terminated

We are happy to send full log files offline.

### Relevant log output

```shell
# Fast Fail case
2026-01-13 02:56:05 error ... BigQueryException: 400 Bad Request ...
2026-01-13 02:56:06 error DestinationReader error:
2026-01-13 02:56:06 info DestinationReader finished.
2026-01-13 02:56:06 info MessageProcessor finished.
2026-01-13 02:56:09 warn ... SourceReader error: java.io.IOException: Broken pipe
2026-01-13 02:56:09 info SourceReader finished.

# Hanging Case
2026-01-13 03:23:21 error ... BigQueryException: 500 Internal Server Error ...
2026-01-13 03:23:21 info DestinationReader finished.
2026-01-13 03:23:21 info MessageProcessor finished.
... # No SourceReader finished log
2026-01-13 05:12:10 info ... SourceReader coroutine state still reading
```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10927

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.