apache / apache/texera

Add workflow-to-python translator

Open
#5,127 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Currently, users have no way to take a workflow they have built in the Texera UI and run it outside of the Texera engine. This feature adds the ability to export any workflow as a standalone Python script that can be viewed in a modal and run locally.

### Proposed Solution or Design

A `WorkflowToPythonTranslator` component is added to the Amber engine. It takes a workflow's logical plan (the DAG of operators), iterates through the operators in topological order, and assembles a standalone Python script. Each operator is assigned a sequential DataFrame variable (df1, df2, ...) based on its position in the DAG.

Each operator descriptor (e.g., `CSVScanSourceOpDesc`) implements the `StandaloneCodeGenerator` trait, which provides a `generateStandaloneCode()` method. This method returns the equivalent Python code for that operator using standard data science libraries, decoupled from the Texera engine. The generated code uses placeholder variable names (`in1df`, `out1df`) which the translator replaces with the actual assigned variable names via `substituteVars()`.

Leaf operators that produce DataFrames are printed at the end of the script so it gives visible output when run locally. Visualization operators (e.g., bar chart) write their output to an HTML file instead.

#### Architecture Diagram
Image

#### BarChart Demo
Workflow with a CSV scan, filter, sort, and bar chart. Shows the workflow result in Texera, exports to Python via the modal, and runs the generated script locally to produce a matching bar chart.

https://github.com/user-attachments/assets/70cadd54-5d0e-4cad-b957-76aab2abee0e

#### DataFrame Demo
Workflow with a CSV scan, filter, and sort. Demonstrates how leaf DataFrame outputs are printed when the script is run locally.

https://github.com/user-attachments/assets/0e546c82-b958-40af-920f-893a2edb6829

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the Amber workflow engine's logical-plan/DAG and operator descriptor entry points, then review the proposed StandaloneCodeGenerator and substituteVars responsibilities. Done means workflows export as standalone Python scripts, leaf DataFrames are printed, and visualization outputs are written to HTML.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.