apache / apache/texera

Mount versioned LakeFS repositories into computing-unit pods (FUSE / GeeseFS)

Open
#6,606 1 comment 0 reactions 0 assignees View on GitHub
feature infra proposal workflow-pods
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Mount a **versioned LakeFS repository directly into a computing-unit pod's file system**, so operator code can read a repository's files from a local path instead of downloading them over HTTP first. LakeFS backs both datasets and models in Texera, so this works for either.

**There is no explicit mount action.** A Python UDF declares a parameter whose value names a model or dataset, and the mount follows from that:

```python
class ProcessTupleOperator(UDFOperatorV2):
def open(self):
model_dir = self.UiParameter("IRIS_MODEL", AttributeType.STRING, value=Resource.MODEL)
self.model = joblib.load(f"{model_dir}/iris.pkl")
```

The parameter is an ordinary string; only where the string comes from differs. The property panel offers that resource's browser instead of a text box, the user picks a **version**, and what the UDF receives at run time is the local directory that version is mounted at. Choosing the version and mounting it are settled when the workflow runs, not while it is being edited.

**Problem it solves:** today, if an operator needs a repository's files (e.g. a large ML model with multiple shards), the options are to stream them through the storage API or download the whole repository into the pod before use. For large repositories (multi-GB models) this is slow, uses a lot of pod disk, and forces a full transfer even when the code only reads part of the data.

### Proposed Solution or Design

Texera stores datasets and models as **LakeFS** repositories on MinIO/S3. LakeFS exposes an **S3-compatible gateway**, so a repository at a specific commit can be mounted as a read-only file system using a FUSE S3 client. We use **[GeeseFS](https://github.com/yandex-cloud/geesefs)** for this.

Key properties:
- **Lazy / on-demand** — the mount transfers zero bytes; file contents are fetched with ranged reads only when the code actually `read()`s them, with read-ahead for throughput.
- **Version-pinned & immutable** — a repository version maps to a LakeFS commit, so the mounted content is content-addressed and never changes underneath the reader.
- **Storage-backend agnostic** — GeeseFS always talks to the LakeFS S3 gateway, so it works identically whether LakeFS is backed by in-cluster MinIO or an external S3 bucket.

**Architecture (Kubernetes) — the mount is performed out-of-pod.** The computing-unit pod runs untrusted user code, so it stays **unprivileged**. The FUSE mount is performed by a per-node **`texera-mounter`** privileged DaemonSet that we own and audit; the resulting read-only mount is exposed back into the CU pod through Kubernetes **mount propagation**, scoped to that CU. This shrinks the privileged surface from "every UDF" to one small reviewed program.

**Authorization (no global credentials in the pod).** The mount reuses the per-user JWT the pod already holds and the authorization file-service already applies to repository reads (`userHasReadAccess`). A **JWT-authenticated S3 proxy** in file-service sits in front of the LakeFS S3 gateway: GeeseFS is given the pod's JWT as its S3 access key, the proxy verifies the JWT, checks the user's read access to the requested repository, and re-signs to LakeFS with the global credentials held **only server-side**. No per-mount credential is minted or stored; file-service holds no per-mount state and stays horizontally scalable.

Image

**Only access-control-service may ask for a mount.** The mounter is the one privileged component — root on every node, listening on a hostPort — so it authenticates its caller with the Kubernetes `TokenReview` API against a token bound to its own audience, and the allow-list holds one identity: access-control-service, which is already where the deployment decides whether a user may act on a computing unit. A computing-unit pod running user code holds no such token, so a mount request forged from a UDF fails regardless of what it asks for.

**End-to-end flow:**
1. A Python UDF names a model or dataset version through a parameter; the user picks the version in the property panel.
2. When the execution starts, the operator's deferred binding resolves that version path to a `repository:commitHash` locator and rewrites the parameter to the directory the version will be readable at.
3. The region scheduler gathers the deduplicated set of locators its operators name and, for each, asks access-control-service to mount it — presenting the computing unit's own user JWT.
4. Access-control-service validates that JWT, confirms the user's access to the computing unit, finds the node its pod runs on, and asks that node's `texera-mounter` to run GeeseFS against the file-service S3 proxy, read-only.
5. The mount propagates into the CU pod, and the UDF opens files under the directory it was handed.

This has been prototyped and validated end-to-end: a ~2 GB sharded PyTorch model in one repository, loaded inside a Python UDF via `torch.load()` from the mounted path, with bit-exact outputs — against both in-cluster MinIO and a remote AWS S3 backing store.

### Implementation / Sub-issues

Delivered as three stacked PRs to `main`, each kept small and self-contained:

- **[x] Out-of-pod mount infrastructure** (#6862 → PR #6866, merged) — the per-node privileged `texera-mounter`, the file-service JWT S3 proxy, and the unprivileged CU pod's mount-propagation wiring. Performs and authorizes a mount; nothing triggers one yet.
- **[ ] Authorize and perform a repository mount** (#6863 → PR #6896) — the access-control-service mount endpoint that the mounter admits, which decides everything the mounter cannot: request shape, the caller's write access to the computing unit, their read access to the repository, and that the commit belongs to it. No user-facing surface.
- **[ ] Name a model or dataset from a Python UDF parameter** (#6895 → PR #6897) — `value=Resource.MODEL` / `Resource.DATASET` in pytexera, the property-panel resource browser, resolving a version path to its repository and commit, the engine client that asks for the mount, the operator's execution-time binding, and mounting once per region in the scheduler.

**Affected Area:** Deployment / Infrastructure, Workflow Engine (Amber), Storage / Metadata, Workflow UI

Contributor guide

Open the contributing guide

Research direction

Start by reading the architecture and end-to-end flow, then inspect the three implementation sub-issues: #6863 and #6895, alongside the delivered infrastructure in #6862. The feature is done when repository versions selected in Python UDF parameters are authorized, mounted read-only, propagated into the computing-unit pod, and usable at execution time; PRs #6896 and #6897 show work already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, python, scala
Domain
backend, cloud, infrastructure, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.