apache / apache/texera

Add Python Virtual Environment Support

Open
#4,296 7 comments 1 reaction 1 assignee Claimed by @SarahAsad23 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Currently, workflows are executed using the system Python environment. This limits users who need custom packages to run their workflows. In addition, some workflows may contain multiple UDF operators that require different dependencies.

The Python Virtual Environment feature addresses this limitation by allowing users to create multiple environments with custom installed packages. Users can then select the desired environment for each UDF operator through the operator’s property panel, enabling different operators within the same workflow to run in different environments.

### Proposed Solution or Design

In the first iteration of our design, environments will be stored on the local volume of each computing unit. The Python UDF operator will include a dropdown menu that allows users to select from the available environments. During execution, when a UDF operator is encountered, the system will spawn a Python process using the Python executable from the selected virtual environment.

Image

### Impact / Priority

(P1)High – significantly improves user experience

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.