Add Python Virtual Environment Support
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
Currently, workflows are executed using the system Python environment. This limits users who need custom packages to run their workflows. In addition, some workflows may contain multiple UDF operators that require different dependencies.
The Python Virtual Environment feature addresses this limitation by allowing users to create multiple environments with custom installed packages. Users can then select the desired environment for each UDF operator through the operator’s property panel, enabling different operators within the same workflow to run in different environments.
### Proposed Solution or Design
In the first iteration of our design, environments will be stored on the local volume of each computing unit. The Python UDF operator will include a dropdown menu that allows users to select from the available environments. During execution, when a UDF operator is encountered, the system will spawn a Python process using the Python executable from the selected virtual environment.
### Impact / Priority
(P1)High – significantly improves user experience
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.