apache / apache/texera

Export the base transform operators as Python

Open
#8,414 0 comments 0 reactions 1 assignee Claimed by @kz930 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Twenty-two more operators implement `StandaloneCodeGenerator`, so a workflow built from anything but a visualization, a source or a model can be exported as a script that runs: the set operations, the joins, the filters and searches, the reshaping operators, the sorts, the samplers, and the control-flow pair.

Each is written against what its executor does rather than against what its name suggests, and the verification runs both and compares them. Some of what that turns up is visible in the code: Distinct keeps the first occurrence of a duplicate because the executor's `LinkedHashSet` does; Filter guards every comparison with `notna`, because a null answers false to all of them but IS NULL and pandas do not agree on `!=`; the samplers seed their generator so a rerun draws the same rows as the run it is being compared with.

`StandaloneHelpers` holds what several operators share, the samplers' generator and the aggregate's rendering, emitted once near the top of a script rather than inlined per operator, so two samplers in one workflow yield one copy.

Four of these operators change what they do, not only how they export. Regex, Substring Search and Unnest String answered a null cell by raising, which a generated script has no way to reproduce and no reason to: nothing in a column matches nothing, and unnests to no rows. Keyword Search states which keywords its query parser will take, which a user had no way to know from the field alone.

Step 10 of 27 in #8325. It needs #8327 for the trait. Two of the behaviour changes are filed separately and close with the same work, #8074 and #7548; the hyperparameter half of #7936 is a part-of rather than a close.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.