Decouple the lifecycle of a result table from that of a computing unit
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
In the current Texera architecture, a computing unit (CU) of a workflow stores the execution results in its local (temporary) file system. The results will disappear when users terminate the computing unit, thus users cannot fetch the results after that. There are several solutions to this problem.
### Solution 1

- Mount a global volume, denoted as "Vg," to all CUs
- When a workflow finishes its execution in a CU, its results are stored in the global volume
This design is unsafe because malicious users could use UDFs to access other users' data in the Vg volume
### Solution 1 + TableService (marked as "Gateway" in the diagram)

- No mounting between CUs and the Vg volume
- The Table Service exposes endpoints for read/write operations on Vg
### Solution 2
- Each user has a volume, e.g. Bob has a volume Vb and Alice has a volume Va
- Mount Vb for all Bob's CUs, and mount Va for all Alice's CUs
This design still has security concerns as each of the Alice's CU's can access the data of her other CUs
### Solution 2 + Table Service (marked as "Gateway" in the diagram)

- No mounting between a user's volume and a user's CUs
- The Table Service prevents CUs of a user from accessing other users' data
We plan to use Solution 2 in the system.
Contributor guide
Assessment
This issue has not been assessed yet.