apache / apache/texera

Stateful operator upstream of a Loop Start silently overwrites loop variables

Open
#7,248 2 comments 0 reactions 1 assignee Claimed by @VuMartin View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Describe the bug

A Loop Start MERGES any unstamped, counter-0 `State` it receives into its loop variables (`LoopStartOperator.process_state` → `self.state.update(state)`). That merge is load-bearing: the back-edge writes the next iteration's variables to the Loop Start's own input-port state URI with the same "no loop" envelope (`_jump_to_loop_start` → `State.to_tuple(0)`), so the runtime cannot distinguish the loop's own state from someone else's.

The side effect is that **any** operator emitting boundary state via the public `produce_state_on_start` / `produce_state_on_finish` API, upstream of or inside a loop containing a Loop Start, silently injects its keys into that loop's variables:

| Emitted key | Effect |
|---|---|
| collides with a loop variable (e.g. the default `i`) | silently overwrites loop control — wrong iteration count, no error |
| `table` | aborts the loop with `_reserved_name_error` at `produce_state_on_finish` |

```
Source ──▶ statefulOp ──(0, "")──▶ LoopStart ──▶ ... ──▶ LoopEnd

└─ self.state.update({"i": ...}) ← clobbers the loop counter
```

### Where it comes from

`operator.py`'s `self.state.update(state)` dates to #5700. It is not created by #6913, but #6913 makes operator-originated boundary state a first-class flow through loops (the Scala side now emits it with an explicit envelope), so the hazard is easier to reach.

Raised by @Xiao-zhen-Liu in review on #6913 (`#discussion_r3696858248`). The Loop Start side is now documented and pinned there (`test_loopstart_merges_unstamped_state_instead_of_forwarding_it`), but the collision itself is undefended.

Note the Loop End side is NOT affected: its inbound loop state is always stamped by the matching Loop Start, so it can tell the two apart and forwards a body operator's boundary state instead of consuming it.

### Possible directions

1. Namespace the loop's own variables on the envelope (a producer tag on `StateFrame`) so a Loop Start can tell its own state from an upstream operator's, and reject/ignore the latter.
2. Reject a merge that would overwrite an existing loop variable, with a message naming the colliding key.
3. Document it as the contract and add an e2e for `Source → statefulOp → LoopStart`.

### Additional context

Related: #6913, #6660, #6661, #5700.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.