anitnilay20 / anitnilay20/thoth

OpenDAL: open files from remote storage (S3, GCS, Azure, HTTP)

Đang mở
#150 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
architecture feature priority:medium size:large
Ngôn ngữ chính
Rust
Star
70
Fork
5
Merge trung bình
10 giờ 33 phút
Pull request đã merge (30 ngày)
2

Mô tả

## Problem

Thoth currently opens files from the local filesystem only. Data teams routinely work with files stored in object storage (S3, GCS, Azure Blob) or accessible via HTTPS. Opening those files today means downloading them manually first.

## Solution

Integrate [OpenDAL](https://github.com/apache/opendal) — a unified storage abstraction layer that supports 40+ storage backends behind a single `Operator` API. One dependency, many backends.

### Supported backends (initial scope)

| Backend | URI scheme | Notes |
|---------|-----------|-------|
| Local FS | `file://` | Replaces current path handling |
| S3 / R2 | `s3://bucket/key` | AWS, Cloudflare R2, MinIO |
| GCS | `gs://bucket/key` | Google Cloud Storage |
| Azure Blob | `az://container/blob` | |
| HTTPS | `https://host/path` | Public URLs |

### User flow

File → Open Remote… (or paste URI into open dialog) → credential prompt if needed → file streams into Thoth exactly like a local file.

DuckDB integration (#148) works transparently: for formats DuckDB reads natively, the host downloads to a temp path and hands it to DuckDB. For plugin-loaded formats, OpenDAL streams bytes through the existing `get-range` path.

### Credential management

Credentials stored per-backend in the OS keychain (same `secure-storage` infrastructure already in place). Users configure once; Thoth remembers.

### Changes

- Add `opendal` crate to `Cargo.toml` with relevant feature flags
- `src/storage/` — `StorageBackend` abstraction wrapping OpenDAL `Operator`
- File open dialog: accept URIs in addition to local paths
- Credential UI: simple key/value form per backend type (access key + secret, service account JSON, etc.)
- Progress indicator for remote downloads (size may be unknown)

## Acceptance criteria

- [ ] Open files from S3, GCS, HTTPS URIs via the open dialog
- [ ] Credentials stored in OS keychain, not plaintext config
- [ ] Progress bar during download (indeterminate if size unknown)
- [ ] Local file path handling unchanged (no regression)
- [ ] DuckDB native-format path works: remote file downloaded to temp, DuckDB scans it
- [ ] Large files: stream into temp file rather than full memory load

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.