Stateful operator upstream of a Loop Start silently overwrites loop variables
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Describe the bug
A Loop Start MERGES any unstamped, counter-0 `State` it receives into its loop variables (`LoopStartOperator.process_state` → `self.state.update(state)`). That merge is load-bearing: the back-edge writes the next iteration's variables to the Loop Start's own input-port state URI with the same "no loop" envelope (`_jump_to_loop_start` → `State.to_tuple(0)`), so the runtime cannot distinguish the loop's own state from someone else's.
The side effect is that **any** operator emitting boundary state via the public `produce_state_on_start` / `produce_state_on_finish` API, upstream of or inside a loop containing a Loop Start, silently injects its keys into that loop's variables:
| Emitted key | Effect |
|---|---|
| collides with a loop variable (e.g. the default `i`) | silently overwrites loop control — wrong iteration count, no error |
| `table` | aborts the loop with `_reserved_name_error` at `produce_state_on_finish` |
```
Source ──▶ statefulOp ──(0, "")──▶ LoopStart ──▶ ... ──▶ LoopEnd
│
└─ self.state.update({"i": ...}) ← clobbers the loop counter
```
### Where it comes from
`operator.py`'s `self.state.update(state)` dates to #5700. It is not created by #6913, but #6913 makes operator-originated boundary state a first-class flow through loops (the Scala side now emits it with an explicit envelope), so the hazard is easier to reach.
Raised by @Xiao-zhen-Liu in review on #6913 (`#discussion_r3696858248`). The Loop Start side is now documented and pinned there (`test_loopstart_merges_unstamped_state_instead_of_forwarding_it`), but the collision itself is undefended.
Note the Loop End side is NOT affected: its inbound loop state is always stamped by the matching Loop Start, so it can tell the two apart and forwards a body operator's boundary state instead of consuming it.
### Possible directions
1. Namespace the loop's own variables on the envelope (a producer tag on `StateFrame`) so a Loop Start can tell its own state from an upstream operator's, and reject/ignore the latter.
2. Reject a merge that would overwrite an existing loop variable, with a message naming the colliding key.
3. Document it as the contract and add an e2e for `Source → statefulOp → LoopStart`.
### Additional context
Related: #6913, #6660, #6661, #5700.
Contributor guide
Assessment
This issue has not been assessed yet.