google / google/symphony-gcp

Pods that fail to start are not deleted by the orchestrator

Open
#11 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Symptom

- Pods are left dangling on the cluster.
- GCPSR resource is incoherent in relation to pod and HostFactory state.
- Timestamps recorded by HostFactory are incoherent to Pod state.

## Reproduction Steps

1) Create a pod spec which will quickly result in a pod state different than "pending", "running" or "succeeded". For example:

```yaml
containers:
- image: busybox
name: test
command:
- "I don't exist"
restartPolicy: "Never"
```
2) Request this machine from the Symphony Host Factory connector.
3) Request the closure of this request (redundant, since request is already considered closed after failure).

## Problem Description

- Pod is created and started, returning a `StartError` state.
- Symphony queries the CLI, checking for the machine state. The CLI returns the state as failed (since it is not "pending", "running" or "succeeded"). This effectively unbound the machines from Symphony, considering the `status` from the `/demandrequests` HostFactory API as `Closed`. Similarly, the status from the `/requests` HostFactory API is `Done with Errors`.
- In consequence, requesting a machine return will not delete the pod, nor the pod will automatically be handled by the operator.
- Furthermore, the GCPSR state remains in the `Pending` status forever. This demonstrates an incoherent state between Symphony and the custom resource.

## Preliminary Conclusion

State tracking mechanisms differ between the operator and the CLI, resulting in diverging states. While the operator only looks for the `ready` transition of the pod `containerStatuses.state`, the CLI queries the pod `phase` field.

## Version

- Repository hash: 700ff4f8ae089919221985a95484416527febbd4

Contributor guide

Open the contributing guide

Research direction

Start by tracing the operator and CLI state handling described in the issue, comparing pod phase with containerStatuses.state for StartError cases. Reproduce the failing pod specification and verify that pod deletion, HostFactory request states, and the GCPSR status remain consistent after the failed start.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.