apache / apache/texera

Per-element runtime failures can produce silent short results

Open
#8,233 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

`MainLoop._process_data_element` catches per-element runtime failures and writes only a worker log. It never updates the exception manager, so failed rows can disappear from a batch while the workflow continues and may report success with a short result.

Before: row processing fails -> worker log only -> workflow can complete with missing rows

After: row processing fails -> exception is reported -> remaining batch rows are attempted -> worker enters exception handling

Expected behavior:

Every per-element runtime failure should use the normal exception reporting path so the workflow cannot silently succeed with missing output rows.

Reproduction evidence:

Process a two-row `DataElement` while making input statistics update raise for each row. Both rows are attempted, but the exception manager remains empty.

Observed on `70c21145887920528d7d5540e3fb790b43e8b759`:

```text
attempted_rows=[0, 1]
logged_failures=2
exception_manager_has_exception=False
```

The focused regression failed with:

```text
assert main_loop.context.exception_manager.has_exception()
E assert False
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`70c21145887920528d7d5540e3fb790b43e8b759`

### Proposed Solution or Design

Report every per-element runtime failure through the normal exception path so the workflow cannot silently succeed with missing output rows.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.