aws / aws/sagemaker-hyperpod-cli
Inference Operator v3.2.0 ignores l2CacheBackend: redis, hardcodes LMCACHE_REMOTE_URL=sagemaker-hyperpod://...
- Dominant language
- Python
- Stars
- 41
- Forks
- 95
- Avg merge
- 23h 5m
- Merged PRs (30d)
- 8
Description
### Environment
- EKS add-on `amazon-sagemaker-hyperpod-inference` **v1.3.0-eksbuild.1** (latest available in `ap-northeast-1`)
- Inference Operator image `hyperpod-inference-operator:v3.2.0`
- Worker image `lmcache/vllm-openai:v0.4.7` (LMCache 0.4.7, vLLM 0.23.0)
- Instance type `ml.g7e.4xlarge`, HyperPod EKS-orchestrated cluster
### What happened
Setting `kvCacheSpec.l2CacheSpec.l2CacheBackend: redis` (with `l2CacheLocalUrl: redis://...:6379`) in the `InferenceEndpointConfig` has **no effect**. The operator instead injects into the worker pod:
```
LMCACHE_REMOTE_URL=sagemaker-hyperpod://$(NODE_IP):9200
LMCACHE_EXTRA_CONFIG={"sagemaker_hyperpod_shared_memory_name": "ai_toolkit_cache"}
```
The `redis` value from the CRD is silently dropped — the operator never emits `LMCACHE_REMOTE_URL=redis://...`. Attempting to override `LMCACHE_REMOTE_URL` via `worker.environmentVariables` does **not** work either: those `LMCACHE_*` entries do not appear in the rendered pod (the operator strips/overrides them).
Because the worker is forced onto the `sagemaker-hyperpod` connector, and that connector requires a host-side `ai-toolkit` daemon (POSIX shared memory `/ai_toolkit_cache` + TCP `:9200`) which is not present on the cluster, LMCache enters degraded mode and the L2 cache never stores anything:
```
LMCache ERROR: Failed to initialize shared memory: [Errno 22] Invalid argument: '/ai_toolkit_cache'
LMCache WARNING: Health check failed: RemoteBackendHealthCheck(sagemaker-hyperpod://:9200)
LMCache WARNING: HealthMonitor: System unhealthy, entering degraded mode
LMCache WARNING: LMCache is unhealthy, skipping store operation
... LMCache hit tokens: 0 (External prefix cache hit rate: 0.0%)
```
### Expected behavior
With `l2CacheBackend: redis` + `l2CacheLocalUrl: redis://...:6379`, the worker's LMCache should be configured with `LMCACHE_REMOTE_URL=redis://...`, connect to the specified Redis endpoint, report healthy, and perform L2 KV-cache store/lookup. `redis` is listed as a supported L2 backend in the docs ([KV cache & intelligent routing](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-caching-routing.html)), so the operator overriding it contradicts the documentation.
### Reproduction
1. Deploy an `InferenceEndpointConfig` with:
```yaml
kvCacheSpec:
enableL1Cache: true
enableL2Cache: true
l2CacheSpec:
l2CacheBackend: redis
l2CacheLocalUrl: redis://..svc.cluster.local:6379
```
2. (Optionally) add `worker.environmentVariables` entries for `LMCACHE_REMOTE_URL` to try to override.
3. Inspect the rendered worker pod:
```
kubectl get pod -o jsonpath='{range .spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' | grep LMCACHE
```
4. Observe `LMCACHE_REMOTE_URL=sagemaker-hyperpod://$(NODE_IP):9200` (not `redis://...`), and the worker logs showing the LMCache unhealthy/degraded loop above.
### Additional question (tieredstorage)
Separately: `l2CacheBackend: tieredstorage` requires the host-side `ai-toolkit` daemon (shm `/ai_toolkit_cache` + `:9200`). On our cluster this daemon is not installed as any DaemonSet, and the add-on configuration schema (`aws eks describe-addon-configuration`) exposes no toggle for it (only `alb, enableCustomServiceAccounts, executionRoleArn, hyperpodClusterArn, jumpstartGatedModelDownloadRoleArn, keda, tlsCertificateS3Bucket`). How is tiered storage meant to be enabled — a cluster-creation flag, and can it be enabled on an existing cluster?
### References
- LMCache SageMaker HyperPod connector (client expecting a pre-existing daemon): https://github.com/LMCache/LMCache/pull/1937
- LMCache backend docs: https://docs.lmcache.ai/kv_cache/storage_backends/sagemaker_hyperpod.html
- AWS KV cache & routing docs: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-caching-routing.html
Contributor guide
Research direction
Start by reproducing the InferenceEndpointConfig and inspect the rendered worker pod with the provided kubectl jsonpath command. Trace how the operator handles l2CacheBackend, l2CacheLocalUrl, and worker.environmentVariables; done means Redis produces LMCACHE_REMOTE_URL=redis://..., the worker reports a healthy LMCache backend, and the tieredstorage deployment path is documented or clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, python, redis
- Domain
- cloud, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100