Python worker can hide batch processing failures
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
The Python worker catches an exception while unpacking a data batch but only writes it to the worker log. It then consumes the remaining rows and can report successful completion with missing output.
Before: a row fails, later rows continue, and the coordinator receives no error.
Expected: the first failure is reported to the coordinator and the worker enters exception pause without consuming later rows.
Reproduction evidence:
1. Send a two row data frame to a Python worker.
2. Make tuple bookkeeping fail on the first row.
3. Observe that both rows are attempted and the exception manager remains empty.
Version and commit evidence:
1.3.0-incubating-SNAPSHOT on main
**Commit Hash (Optional)**
50321e403c82df299a13deb50a7f9849dd93bdba
**Relevant log output**
RuntimeError: statistics backend unavailable
### Proposed Solution or Design
Expected: the first failure is reported to the coordinator and the worker enters exception pause without consuming later rows.
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.