Fully Support AWS S3 for Texera, Store All Data in One Place

In our current design, to store all the data in the whole workflow (execution logs, intermediate data, user inputs, etc), we can only use minio as the storage place. That is not effieient, both in terms of cost and time. Using minio as storage can be really cost-consuming because it is a serveless design, which means that it has to keep on running on a server and costs huge amount of money. Also using minio is not very scalable and universal, especially when users are from finance, politics or other backgrounds where data is highly sensitive and users prefer to keep the data on their own.

In response to all those drawbacks, we plan to move the storage place from minio to AWS S3, which is much more cost-effective, highly secured and confedential. Althouth minio uses AWS S3 API, we are still uncertain about the possibility and efficiency.

We start with first moving the whole data into S3, because currently our design is to treat all storage as a single piece. No matter what kinds of data will finally come to the same destination. If we want to further differentiate user data from our barin layer data, we need to further modify our code deeper.

Start by reviewing the current MinIO-based storage design and how workflow data such as execution logs, intermediate data, and user inputs are handled. Determine the feasibility and efficiency of using AWS S3 through the existing S3 API compatibility. Done means the whole workflow’s data is stored in S3 instead of MinIO, with the storage approach documented.

Written by the indexing model from the issue text.