agent-substrate / agent-substrate/substrate

[Bug]: Actor remains in resuming state on runsc error

Abierto
#1,518 0 comentarios 0 reacciones 1 asignado Reclamado por @juli4n Ver en GitHub
area/api-machinery kind/bug
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

### What happened?

When launching an actor and runsc fails (e.g. `grpc ResumeActor: rpc error: code = Internal desc = while restoring "glutton" application container: while running `runsc restore`: exit status 128`), the actor remains stuck in resuming and assigned to a worker, removing that worker from the pool of available workers.

### Expected Behavior

Would expect the actor to go into crashed state or similar and the worker to be freed.

### Steps to Reproduce

You can use this patch to force a runsc error whenever resuming an actor:

https://github.com/maxsmythe/substrate/commit/6e27c956e326e66b08afa54a3d186fa8dca993bd

shell log of behavior in action:

```
smythe@smythe63:~/dev/substrate$ kubectl ate create actor probe-2 -a benchmark-workloads --template-ref sleep
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_SUSPENDED 1 0s
smythe@smythe63:~/dev/substrate$ kubectl ate resume actor probe-2 -a benchmark-workloads
Error: failed to resume actor: rpc error: code = Internal desc = while restoring pause container: while running `runsc restore`: exit status 128
smythe@smythe63:~/dev/substrate$ kubectl ate get actor probe-2 -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 30s
smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 4m12s
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 52s
smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1

smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 15m
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 12m

smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1
```

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

patched version of main (to force error)

### Kubernetes Version & Environment

_No response_

### Host OS & Architecture

_No response_

### Relevant Logs and Diagnostic Output

```shell

```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.