A table operator handed a port that carried no rows loses every column
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
A port can finish having carried no rows. An upstream filter that matches nothing still ends its channel, and `DataProcessor` calls `on_finish` either way. `TableOperator` then builds its table out of no tuples, and since `Table.from_tuple_likes` reads the column names off the tuples it is given, the operator is handed a DataFrame with no columns at all rather than the declared columns with no rows under them.
Every table operator that names one of its own columns raises `KeyError` on that column. Sort is the plainest case, but this reaches the visualization operators and any user-written `UDFTableOperator` as well. The expected behaviour is the one the schema already describes: the port's declared columns, empty.
### How to reproduce?
Build a workflow of CSV File Scan, then Filter with a condition no row satisfies, then Sort on one of the scanned columns, and run it. Sort fails with `KeyError` naming the sort column, even though that column exists on the link.
The same thing shows without the engine, from the Python worker's own models. `Table(Table.from_tuple_likes([]))` reports `columns: []`, and calling `sort_values(by=["x"])` on it raises `KeyError: 'x'`. Building the frame from the port's schema instead, through `pyarrow.Table.from_pylist([], schema=schema.as_arrow_schema()).to_pandas()`, gives columns `['x', 'name']` with dtypes `int32` and `object`, and the same sort succeeds with shape `(0, 2)`.
Part of #8325: the export's verification runs each operator on an empty table too, and a Python operator's engine-side run cannot be compared until this is fixed. The bug itself is not about the export, and reaches Sort, the visualization operators and any user-written `UDFTableOperator` in an ordinary run.
### Version/Branch
1.4.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
1fbd34627
### Relevant log output
```shell
KeyError: 'x'
```
Contributor guide
Assessment
This issue has not been assessed yet.