apache / apache/texera

Import files from Google Drive into Texera datasets

Open
#5,634 2 comments 0 reactions 1 assignee Claimed by @eugenegujing View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

## Background

Sub-task of #5242 (external data source import and export). Per the discussion in #4240, the import direction is handled as a separate effort from the Google Drive export work (#5250 / #5251 / #5252 by @Sentiaus).

This task adds the **import** direction for the first provider, Google Drive, along with a minimal provider abstraction so additional providers (Dropbox — see sibling sub-issue; Box later) can plug in, as suggested by @xuang7.

## Design principles (following @aicam's review decisions on the export flow in #4240)

1. **No token persistence** — the user authorizes each import; the frontend obtains a one-time OAuth access token, passes it to the backend with the import request, and the backend discards it after the transfer. Nothing is ever stored in the DB.
2. **Backend streaming** — the backend streams the file directly from the Google Drive API into dataset storage (LakeFS/S3), reusing the existing multipart upload pipeline in `DatasetResource`. The file never round-trips through the browser.

## Scope choice: Google Picker + `drive.file` only

The frontend uses the official **Google Picker** for file selection, requesting only the **`drive.file`** scope:

- `drive.file` is a **non-sensitive** scope: deployments need **no Google restricted-scope security verification**, in any status (Testing or Production).
- Google enforces at the permission layer that the app can only access files the user explicitly picked in the Picker — the app never lists or sees the rest of the user's Drive.
- The consent screen reduces to a single grant ("access only the specific files you use with this app"), which also avoids partial-consent failure modes (e.g., an in-app file browser turning up empty when a user denies the broad scope).

The alternative — rendering a Drive file tree inside Texera — would require the restricted `drive.readonly` scope ("see and download all your Google Drive files") and a weeks-long Google security review per public deployment, for a UX that is effectively identical in the single-file import case. The provider interface still exposes a `listFiles` capability so in-app browsing can be added later if the community accepts that cost (the Dropbox provider will use it, since Dropbox has no comparable review process).

## Proposed changes

- **Backend (file-service):**
- A small provider interface (e.g. `CloudStorageImportProvider`: list files / open download stream) with a Google Drive implementation (`GET /drive/v3/files/{fileId}?alt=media` streaming download using the one-time bearer token).
- A new endpoint on `DatasetResource`, e.g. `POST /dataset/{did}/import-from-cloud`, taking `{provider, accessToken, fileId, fileName}`; streams the file into the dataset's LakeFS repo and stages/commits it like a normal upload, with the same dataset write-permission checks as the existing upload endpoint.
- Provider configuration (OAuth client ID, Picker API key) via env vars, mirroring how Google login is configured today (`UserSystemConfig.googleClientId`).
- **Frontend:**
- An "Import from cloud" entry next to the existing file uploader on the dataset page.
- One-time authorization via the Google Identity Services token client (`drive.file` scope), then the Google Picker for selection, restricted with `setSelectableMimeTypes` to formats Texera datasets accept (csv, json, parquet, text, etc.), so unsupported types (videos, native Google Docs) are filtered out at selection time instead of failing after import.
- A simple in-progress indicator while the backend streams the file into storage.

## Out of scope (future iterations)

- **Folder / bulk import** — folder-level access would require the restricted `drive.readonly` scope and Google's security verification (weeks-to-months review); deferred until the need is validated.
- **In-app Drive file browsing** — same `drive.readonly` requirement; see scope-choice section above.
- **Background/async job handling for very large files** (the 100 GB–1 TB scale discussed in #4240); the MVP imports synchronously within the request.
- **Native Google Docs/Sheets/Slides export conversion** — only binary/regular files are imported in the MVP.

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [x] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.