apache / apache/texera

A table operator handed a port that carried no rows loses every column

Open
#8,487 0 comments 0 reactions 1 assignee Claimed by @kz930 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

A port can finish having carried no rows. An upstream filter that matches nothing still ends its channel, and `DataProcessor` calls `on_finish` either way. `TableOperator` then builds its table out of no tuples, and since `Table.from_tuple_likes` reads the column names off the tuples it is given, the operator is handed a DataFrame with no columns at all rather than the declared columns with no rows under them.

Every table operator that names one of its own columns raises `KeyError` on that column. Sort is the plainest case, but this reaches the visualization operators and any user-written `UDFTableOperator` as well. The expected behaviour is the one the schema already describes: the port's declared columns, empty.

### How to reproduce?

Build a workflow of CSV File Scan, then Filter with a condition no row satisfies, then Sort on one of the scanned columns, and run it. Sort fails with `KeyError` naming the sort column, even though that column exists on the link.

The same thing shows without the engine, from the Python worker's own models. `Table(Table.from_tuple_likes([]))` reports `columns: []`, and calling `sort_values(by=["x"])` on it raises `KeyError: 'x'`. Building the frame from the port's schema instead, through `pyarrow.Table.from_pylist([], schema=schema.as_arrow_schema()).to_pandas()`, gives columns `['x', 'name']` with dtypes `int32` and `object`, and the same sort succeeds with shape `(0, 2)`.

Part of #8325: the export's verification runs each operator on an empty table too, and a Python operator's engine-side run cannot be compared until this is fixed. The bug itself is not about the export, and reaches Sort, the visualization operators and any user-written `UDFTableOperator` in an ordinary run.

### Version/Branch

1.4.0-incubating-SNAPSHOT (main)

### Commit Hash (Optional)

1fbd34627

### Relevant log output

```shell
KeyError: 'x'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.