vllm-project / vllm-project/aibrix

KubernetesJob batch worker does not receive Redis password → fails on an authenticated metastore

Open
#2,262 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🐛 Describe the bug

## Summary
For the `KubernetesJob` (self-hosting / SingleJobRunner) batch path, the worker pod
inherits the metadata service's Redis metastore config via `build_metastore_env()`,
but the **password is never propagated**. On any Redis that requires auth, the worker
connects unauthenticated and metastore I/O fails — so the batch never produces results.

## Where
`python/aibrix/aibrix/batch/manifest/storage_env.py` — `build_metastore_env()` / `_redis_env()`
emit `REDIS_HOST`, `REDIS_PORT`, `REDIS_DB` only. There is an explicit comment:
`# Password cannot be passed in env; set it via a secret ref.` — but no secret ref is
actually wired, so the password is simply dropped.

## Impact
- Works today only because our dev Redis (`svc/redis`) has no auth.
- Any production deployment with `REDIS_PASSWORD` / `STORAGE_REDIS_PASSWORD` set will have
workers fail to read/write the metastore → batches stuck / 0 results.
- Affects `KubernetesJob` only. The `Kubernetes` (Deployment) path does its I/O in the
metadata process and is unaffected.

## Repro
1. Run MDS against a password-protected Redis metastore.
2. Submit a batch with `extra_body.aibrix.compute.provider = "KubernetesJob"`.
3. Worker pod connects to Redis without the password → metastore ops fail.

## Proposed fix (not implemented)
Propagate the Redis password to the worker pod via a Kubernetes `secretKeyRef`, e.g. a
`WORKER_REDIS_PASSWORD_SECRET` (name/key) the renderer turns into a `valueFrom.secretKeyRef`
`REDIS_PASSWORD` env — never a literal value. Mirror the existing
`WORKER_REDIS_HOST` / `WORKER_STORAGE_AWS_ENDPOINT_URL` override pattern.

## Acceptance criteria
- [ ] A `KubernetesJob` batch completes end-to-end against an authenticated Redis.
- [ ] The password is delivered via a Secret, not a literal pod-spec env value.
- [ ] Deployment path behavior unchanged.

_Found during the batch-refactor review (branch `jiaxin/batch-refactor-cleanup`)._

### Steps to Reproduce

N/A

### Expected behavior

N/A

### Environment

nightly

Contributor guide

Open the contributing guide

Research direction

Start in python/aibrix/aibrix/batch/manifest/storage_env.py by reading build_metastore_env() and _redis_env(), then trace the renderer handling WORKER_REDIS_HOST and WORKER_STORAGE_AWS_ENDPOINT_URL overrides. Run the KubernetesJob batch path against authenticated Redis and verify the password arrives through a Secret reference, the batch completes, and the Deployment path is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python, redis
Domain
backend, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.