aws-samples / aws-samples/sample-spot-balancer-spark-eks

SpoBalancer pod not starting properly

Open
#2 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi, I've followed instructions to deploy spot-balancer on EKS AutoMode Cluster and when I'm checking spot-balancer-webhook logs, I see following errors. Any idea what this can be related to?

```shell
kubectl -n spark-tools logs deploy/spot-balancer-webhook -c spot-balancer --tail=100
AwsEksResourceDetector failed: HTTP Error 403: Forbidden
AwsEc2ResourceDetector failed: timed out
[2026-05-22 03:45:08 +0000] [1] [INFO] Starting gunicorn 26.0.0
[2026-05-22 03:45:08 +0000] [1] [INFO] Listening at: https://0.0.0.0:8443 (1)
[2026-05-22 03:45:08 +0000] [1] [INFO] Using worker: sync
[2026-05-22 03:45:08 +0000] [21] [INFO] Booting worker with pid: 21
[2026-05-22 03:45:08 +0000] [1] [ERROR] Control server error: [Errno 30] Read-only file system: '/home/appuser/.gunicorn'
2026-05-22 03:45:08,693 INFO Starting reconcile of Redis state from K8s
2026-05-22 03:45:08,693 INFO Reconcile worker started (interval=1800s)
2026-05-22 03:45:08,704 ERROR Reconcile error: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '449ca280-3a06-4ef7-b1ad-9292204780ca', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 22 May 2026 03:45:08 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the EKS AutoMode deployment and reviewing the spot-balancer-webhook logs shown in the issue. Investigate the reported AWS detector permissions, Kubernetes 401 response, and read-only /home/appuser/.gunicorn path, then identify the configuration or permissions change that allows the pod to start and reconcile Redis state successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, python, redis
Domain
authentication, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.