flyteorg / flyteorg/flyte

AppEnvironment (Flyte Apps) has no supported mechanism for mounting a pre-existing PersistentVolumeClaim

Open
#7,702 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 14h
Merged PRs (30d)
120

Description

**_What ?_**
Flyte tasks support mounting PVCs via PodTemplate overrides (V1PodSpec, V1Volume, V1VolumeMount), but flyte.app.AppEnvironment and by extension plugin environments built on top of it, e.g. VLLMAppEnvironment — currently has no equivalent. There is no volumes=/volume_mounts= field on AppEnvironment, and attempting to pass a raw pod-spec/K8sPod payload override at deploy time fails with
`
failed to build KService for app : K8sPod app payload is not yet supported`

**_Why ?_**
We're serving a large (~tens of GB) vLLM-backed model via VLLMAppEnvironment. The model is too large to bake into the container image practically (image bloat, slow build/push/pull cycles), and we'd like fast, warm-start-friendly access to model weights already staged on a PVC on our cluster, rather than re-downloading/streaming from object storage on every cold start.

VLLMAppEnvironment currently only supports two model sources:

model_hf_path - download from Hugging Face at startup
model_path (S3-style remote path) - either download-then-load, or stream safetensors directly to GPU via the custom FlyteModelLoader (--load-format flyte-vllm-streaming)

Neither of these lets us point at a volume that's already populated and mounted, which is the standard pattern for serving large models on Kubernetes outside of Flyte.

**_What we tried_**

* Passing a flyte.PodTemplate/raw pod-spec override analogous to what's needed for regular Flyte tasks (see flyteorg/flyte#2754 and our own workaround for imagePullSecrets on tasks) — rejected by the backend with K8sPod app payload is not yet supported.
Checked AppEnvironment's public dataclass fields (name, port, image, resources, env_vars, secrets, parameters, scaling, links) — none expose volumes.
* Considered subclassing AppEnvironment (as VLLMAppEnvironment does) to add this ourselves — not possible from the public SDK surface, since volume mounting requires a field in the underlying App spec sent to the backend, not something a dataclass subclass can add on its own.

Proposed API (open to alternatives)

Something along the lines of:
```

python
from flyte.app import Volume, VolumeMount

app = VLLMAppEnvironment(
name="my-vllm-app",
model_path="/mnt/models/my-model", # local path, not remote
model_id="my-model",
volumes=[
Volume(
name="model-storage",
persistent_volume_claim="my-model-pvc",
),
],
volume_mounts=[
VolumeMount(name="model-storage", mount_path="/mnt/models", read_only=True),
],
...
)
```

Or, if a lower-level but still-supported (i.e., not the rejected raw K8sPod payload) mechanism already exists internally, documentation pointing to it would also resolve this.

**_Why this matters beyond our specific case_**

Any App serving large models/datasets that are managed independently of the container image (updated on a different cadence, shared across multiple apps, staged by a separate data pipeline) hits this same gap. PVC support for Apps would bring them to parity with what's already supported for Flyte tasks.

Contributor guide

Open the contributing guide

Research direction

Trace AppEnvironment and VLLMAppEnvironment from the public Python SDK into the App spec and backend KService-building path; first compare task PodTemplate PVC handling with the rejected K8sPod app payload. Define how volumes and volume_mounts should reach the Kubernetes app configuration, then verify that an App can mount an existing PersistentVolumeClaim and that existing task and app deployment behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, python
Domain
api, backend, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.