flyteorg / flyteorg/flyte-agent-plugins
flyte-deploy-kind: local kind + R2 deploy needs task-pod storage creds, runs.storagePrefix override, and console-origin fix
- Dominant language
- Python
- Stars
- 2
- Forks
- 4
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 11
Description
## Summary
While deploying a local **kind**-based Flyte v2 (`flyte-binary` v2.0.27) cluster on macOS using the [`flyte-deploy-kind`](https://github.com/flyteorg/skills/tree/main/plugins/flyte-deploy-kind) skill, the control plane came up healthy and the API (`ListProjects`) worked, but **task execution and the web console did not work out of the box**. Each issue was diagnosable but not covered by the skill runbook. Filing to suggest concrete additions.
Deployment shape: kind on the local machine + Supabase Postgres (session pooler) + Cloudflare R2 object store.
---
## Issues faced (in the order they surfaced)
### 1. Web console can't reach the API — console and API are separate origins
Port-forwarding `flyte-console` directly (`:8080`) serves only the Next.js SPA. The console frontend resolves its API base URL as `NEXT_PUBLIC_ADMIN_API_URL || "/"`, and that env var is unset, so the browser calls the API at the **same origin** it was served from (`localhost:8080/flyteidl2.*`) — which 404s because `:8080` doesn't serve the API. Result: run pages load blank / "connection refused" on the URL the SDK prints (`http://localhost:8080/v2/...`).
**What actually works:** put console + API behind **one origin**. Traefik (installed by the auth flow) with two IngressRoutes — `PathPrefix('/flyteidl2.')` → `flyte-http:8090` (scheme `h2c`) and `/` → `flyte-console:80` — then `kubectl -n traefik port-forward service/traefik 8080:80` and open `http://localhost:8080/v2`.
### 2. Task pods have no object-store credentials → fall back to AWS IMDS
Tasks fail with:
```
OSError: Generic S3 error: Error performing PUT http://169.254.169.254/latest/api/token ... HTTP error: error sending request
```
The task-side SDK (obstore) reads static creds from `FLYTE_AWS_ENDPOINT` / `FLYTE_AWS_ACCESS_KEY_ID` / `FLYTE_AWS_SECRET_ACCESS_KEY` (`flyte/storage/_config.py`, `S3._KEY_ENV_VAR_MAPPING`). With none set it uses the default AWS credential chain and hits the EC2 metadata endpoint `169.254.169.254`. The chart's `storage.*` values configure the **control plane** but do not propagate credentials to **task pods**.
**Fix:** inject them via `configuration.inline.plugins.k8s.default-env-vars`. Caveat: this list **replaces** the chart default, so the control-plane vars must be repeated:
```yaml
configuration:
inline:
plugins:
k8s:
default-env-vars:
- _U_EP_OVERRIDE: "flyte-http.flyte:8090"
- _U_INSECURE: "true"
- _U_USE_ACTIONS: "1"
- FLYTE_AWS_ENDPOINT: "https://.r2.cloudflarestorage.com"
- FLYTE_AWS_ACCESS_KEY_ID: ""
- FLYTE_AWS_SECRET_ACCESS_KEY: ""
```
### 3. Task I/O written to the wrong bucket → 403 AccessDenied
After creds were fixed, tasks then failed with:
```
403 Forbidden AccessDenied ... PUT https://.r2.cloudflarestorage.com/flyte-data/.../error.pb
```
`runs.storagePrefix` defaults to `s3://flyte-data` (a bucket that doesn't exist), so all task input/output/error writes go to the wrong bucket. `storage.metadataContainer` / `userDataContainer` only set the dataproxy container — **not** this prefix.
**Fix:**
```yaml
configuration:
inline:
runs:
storagePrefix: s3://
```
### 4. Minor / papercuts
- **`helm upgrade` kills the `flyte-http` port-forward.** Every upgrade rolls the flyte pod, dropping `kubectl port-forward service/flyte-http 8090:8090`; the SDK then reports "Flyte system is currently unavailable." Worth calling out to restart it after each upgrade.
- **Console vs SDK use different ports.** Browser → `:8080` (Traefik unified origin), SDK → `:8090` (`flyte-http`). Easy to conflate; the run URL advertises `:8080` while the SDK config points at `:8090`.
- **Default SDK config port.** A generated project-local `.flyte/config.yaml` pointed at `localhost:8080` for `admin.endpoint`; the API is on `8090`.
---
## Suggested skill improvements
1. **Add a "Storage for task pods" step** for any S3-compatible store (R2/MinIO), documenting the three `FLYTE_AWS_*` env vars and the `default-env-vars` list-replacement caveat. This is required for tasks to run at all — not optional.
2. **Add `runs.storagePrefix` override** to the storage step (or derive it from `metadataContainer`), and explain it's distinct from `metadataContainer`/`userDataContainer`.
3. **Document console access without auth.** The base (no-auth) path currently leaves the console unreachable for run pages; add the Traefik unified-origin recipe (or set `NEXT_PUBLIC_ADMIN_API_URL`) so `/v2` run pages load. Today Traefik only appears in the optional auth section.
4. **Note the port-forward lifecycle** — that `helm upgrade` drops the `flyte-http` forward, and that browser (`:8080`) and SDK (`:8090`) use different forwards.
5. **Consider a values snippet** that bundles items 1–2 into the R2/S3 example so the base deployment is task-ready, not just API-ready.
Happy to open a PR against the skill with these additions if useful.
_Environment: macOS, Docker Desktop, kind, flyte-binary v2.0.27, flyte SDK 2.5.8, Cloudflare R2 + Supabase (session pooler)._
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the flyte-deploy-kind skill runbook, especially its storage and optional auth sections, and compare them with the reported R2, task-pod, console, and port-forward behavior. Add the task-pod environment variables, runs.storagePrefix example, unified-origin console access, and port-forward lifecycle notes; done means the base runbook covers a task-ready deployment and working console paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, python
- Domain
- cloud, devops, documentation, infrastructure
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100