vllm-project / vllm-project/aibrix
KubernetesJob batch worker does not receive Redis password → fails on an authenticated metastore
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🐛 Describe the bug
## Summary
For the `KubernetesJob` (self-hosting / SingleJobRunner) batch path, the worker pod
inherits the metadata service's Redis metastore config via `build_metastore_env()`,
but the **password is never propagated**. On any Redis that requires auth, the worker
connects unauthenticated and metastore I/O fails — so the batch never produces results.
## Where
`python/aibrix/aibrix/batch/manifest/storage_env.py` — `build_metastore_env()` / `_redis_env()`
emit `REDIS_HOST`, `REDIS_PORT`, `REDIS_DB` only. There is an explicit comment:
`# Password cannot be passed in env; set it via a secret ref.` — but no secret ref is
actually wired, so the password is simply dropped.
## Impact
- Works today only because our dev Redis (`svc/redis`) has no auth.
- Any production deployment with `REDIS_PASSWORD` / `STORAGE_REDIS_PASSWORD` set will have
workers fail to read/write the metastore → batches stuck / 0 results.
- Affects `KubernetesJob` only. The `Kubernetes` (Deployment) path does its I/O in the
metadata process and is unaffected.
## Repro
1. Run MDS against a password-protected Redis metastore.
2. Submit a batch with `extra_body.aibrix.compute.provider = "KubernetesJob"`.
3. Worker pod connects to Redis without the password → metastore ops fail.
## Proposed fix (not implemented)
Propagate the Redis password to the worker pod via a Kubernetes `secretKeyRef`, e.g. a
`WORKER_REDIS_PASSWORD_SECRET` (name/key) the renderer turns into a `valueFrom.secretKeyRef`
`REDIS_PASSWORD` env — never a literal value. Mirror the existing
`WORKER_REDIS_HOST` / `WORKER_STORAGE_AWS_ENDPOINT_URL` override pattern.
## Acceptance criteria
- [ ] A `KubernetesJob` batch completes end-to-end against an authenticated Redis.
- [ ] The password is delivered via a Secret, not a literal pod-spec env value.
- [ ] Deployment path behavior unchanged.
_Found during the batch-refactor review (branch `jiaxin/batch-refactor-cleanup`)._
### Steps to Reproduce
N/A
### Expected behavior
N/A
### Environment
nightly
Contributor guide
Research direction
Start in python/aibrix/aibrix/batch/manifest/storage_env.py by reading build_metastore_env() and _redis_env(), then trace the renderer handling WORKER_REDIS_HOST and WORKER_STORAGE_AWS_ENDPOINT_URL overrides. Run the KubernetesJob batch path against authenticated Redis and verify the password arrives through a Secret reference, the batch completes, and the Deployment path is unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, redis
- Domain
- backend, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100