apache / apache/texera

Python worker can hide batch processing failures

Open
#8,297 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

The Python worker catches an exception while unpacking a data batch but only writes it to the worker log. It then consumes the remaining rows and can report successful completion with missing output.

Before: a row fails, later rows continue, and the coordinator receives no error.

Expected: the first failure is reported to the coordinator and the worker enters exception pause without consuming later rows.

Reproduction evidence:

1. Send a two row data frame to a Python worker.
2. Make tuple bookkeeping fail on the first row.
3. Observe that both rows are attempted and the exception manager remains empty.

Version and commit evidence:

1.3.0-incubating-SNAPSHOT on main

**Commit Hash (Optional)**
50321e403c82df299a13deb50a7f9849dd93bdba

**Relevant log output**

RuntimeError: statistics backend unavailable

### Proposed Solution or Design

Expected: the first failure is reported to the coordinator and the worker enters exception pause without consuming later rows.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.