A workflow cannot be taken out of Texera as a script that runs on its own
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
### Proposed Solution or Design
An operator says how it reads outside the engine by implementing a `StandaloneCodeGenerator` trait, returning a block of pandas that names its inputs and outputs by position: `in1df`, `in2df`, `out1df`. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.
The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
### The set, in order
1. [x] #8407, the export itself: the trait and the translator. PR #8327
2. [x] #8501, the first five operators, and the endpoint that serves the script. PR #8502
3. [ ] #8408, running an operator through the engine. PR #8356
4. [ ] #8409, running a Python operator the way the engine runs it. PR #8357
5. [ ] #8410, running the script the export produces. PR #8358
6. [ ] #8411, comparing the two tables the paths produced. PR #8359
7. [ ] #8508, comparing the two charts the paths drew. PR #8509
8. [ ] #8412, the canonical table every operator reads. PR #8360
9. [ ] #8413, the configurations a schema cannot describe. PR #8363
10. [ ] #8414, the operators that work on text. PR #8340
11. [ ] #8505, the relational and sampling operators. PR #8506
12. [ ] #8415, sources. PR #8341
13. [ ] #8416, visualizations: hierarchies and graphs. PR #8344
14. [ ] #8417, visualizations: charts over a table's columns. PR #8365
15. [ ] #8418, visualizations: tables, ranges and embedded documents. PR #8366
16. [ ] #8419, visualizations: charts drawn on a coordinate system. PR #8346
17. [ ] #8503, visualizations: charts drawn over a field or a sequence. PR #8504
18. [ ] #8420, the four advanced trainers. PR #8348
19. [ ] #8421, the Machine Learning Scorer. PR #8367
20. [ ] #8422, the estimators and the Hugging Face models. PR #8368
21. [ ] #8423, deriving a configuration from an operator's schema. PR #8361
22. [ ] #8424, the sweep that turns one configuration into the variants a parity run needs. PR #8388
23. [ ] #8425, running every operator both ways and reporting what it did. PR #8362
24. [ ] #8426, the source operators, and the coverage report. PR #8364
25. [ ] #8427, the editor: the export button. PR #8350
26. [ ] #8510, the editor: the property panel reads the schema's constraints. PR #8507
27. [ ] #8458, the guide a contributor reads before adding an operator. PR #8459
Each is sized to be read in one sitting. Six entries were split for growing past that: the export itself into the trait and the operators that use it, the text operators from the relational ones, the coordinate-system charts from the domain ones, the table comparison from the chart comparison, the editor's export button from the property panel, and deriving a configuration into the entry that produces one and the entry that sweeps it into variants. The export comes first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once — those come after the families so their assertions hold as written — and the editor last. This issue stays open until the guide lands.
### Affected Area
Workflow Engine (Amber), Workflow UI
Contributor guide
Assessment
This issue has not been assessed yet.