apache / apache/texera

Add A Python-Notebook-Migration Tool

Open
#4,301 2 comments 0 reactions 1 assignee Claimed by @zyratlo View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Document that tracks all related issues/PRs
https://docs.google.com/document/d/1BU4YxAgmWPmBE90UdIcZ38T8RAQ0q7voxPjrNwo8-3I/edit?usp=sharing

### Feature Summary

Currently, users who have existing code outside of Texera and want to migrate that code to Texera must create a workflow from scratch. This can take a long time to do depending on the complexity of the code. This tool aims to reduce the amount of time needed migrating to Texera by utilizing large language models to migrate Jupyter Notebooks to Texera workflows.

### Proposed Solution or Design
The following architecture diagrams and demo videos demonstrate the proposed tool.

#### Architecture Diagram

Image

Image

Image

#### Demo Videos

1. User uploads a Jupyter Notebook using the new tool button. They can select any available model (depending on the LiteLLM configuration) and provide their API key. In the demo video, the API key is stored as a system environment variable, so it is left blank. After the user submits, they will need to wait until the LLM responds.

https://github.com/user-attachments/assets/d698dc40-939f-49a9-9439-de6ba3f11ca5

2. After the LLM responds, the workspace will reload with the generated workflow and the Jupyter popup.

https://github.com/user-attachments/assets/b471b334-2b42-46df-ae37-f834816f635f

3. The user can edit the workflow however they want. To assist with this process, when the user clicks on a workflow operator, it will highlight the corresponding notebook cell used to generate that operator, and vice versa.

https://github.com/user-attachments/assets/49ab77a2-0766-49ab-9faf-51b3488cfd03

4. The user can minimize the Jupyter panel and reopen it, and can also delete the notebook associated with the workflow.

https://github.com/user-attachments/assets/bc7c1b62-298d-4001-8de6-9bd28a43dec4

5. When the user reopens a workflow that was generated from a notebook, it will also reopen the notebook.

https://github.com/user-attachments/assets/ac94b8ba-cfb1-4c4e-9397-16779e2ec46a

### Related Discussions
Discussion #4590

### Impact / Priority

(P2)Medium – useful enhancement

### Affected Area

Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.