apache / apache/texera

Decouple the lifecycle of a result table from that of a computing unit

Open
#3,381 0 comments 0 reactions 2 assignees Claimed by @aicam View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

In the current Texera architecture, a computing unit (CU) of a workflow stores the execution results in its local (temporary) file system. The results will disappear when users terminate the computing unit, thus users cannot fetch the results after that. There are several solutions to this problem.

### Solution 1

![Image](https://github.com/user-attachments/assets/880c2765-769f-4ff9-b9cb-18b773d3a2c5)

- Mount a global volume, denoted as "Vg," to all CUs
- When a workflow finishes its execution in a CU, its results are stored in the global volume

This design is unsafe because malicious users could use UDFs to access other users' data in the Vg volume

### Solution 1 + TableService (marked as "Gateway" in the diagram)
![Image](https://github.com/user-attachments/assets/e0997533-095d-493c-82a2-a5de2dd58393)

- No mounting between CUs and the Vg volume
- The Table Service exposes endpoints for read/write operations on Vg

### Solution 2

- Each user has a volume, e.g. Bob has a volume Vb and Alice has a volume Va
- Mount Vb for all Bob's CUs, and mount Va for all Alice's CUs

This design still has security concerns as each of the Alice's CU's can access the data of her other CUs

### Solution 2 + Table Service (marked as "Gateway" in the diagram)

![Image](https://github.com/user-attachments/assets/6e949845-4d94-4c82-b5f6-020f030e1077)

- No mounting between a user's volume and a user's CUs
- The Table Service prevents CUs of a user from accessing other users' data

We plan to use Solution 2 in the system.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.