agent-substrate / agent-substrate/substrate

[Bug]: Actor remains in resuming state on runsc error

未关闭
#1,518 0 条评论 0 个 reaction 已指派 1 人 已被 @juli4n 认领 在 GitHub 查看
area/api-machinery kind/bug
主要语言
Go
星标
1.8k
派生
316
平均合并
2 天 43 分钟
30 天内合并 PR
287

描述

### What happened?

When launching an actor and runsc fails (e.g. `grpc ResumeActor: rpc error: code = Internal desc = while restoring "glutton" application container: while running `runsc restore`: exit status 128`), the actor remains stuck in resuming and assigned to a worker, removing that worker from the pool of available workers.

### Expected Behavior

Would expect the actor to go into crashed state or similar and the worker to be freed.

### Steps to Reproduce

You can use this patch to force a runsc error whenever resuming an actor:

https://github.com/maxsmythe/substrate/commit/6e27c956e326e66b08afa54a3d186fa8dca993bd

shell log of behavior in action:

```
smythe@smythe63:~/dev/substrate$ kubectl ate create actor probe-2 -a benchmark-workloads --template-ref sleep
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_SUSPENDED 1 0s
smythe@smythe63:~/dev/substrate$ kubectl ate resume actor probe-2 -a benchmark-workloads
Error: failed to resume actor: rpc error: code = Internal desc = while restoring pause container: while running `runsc restore`: exit status 128
smythe@smythe63:~/dev/substrate$ kubectl ate get actor probe-2 -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 30s
smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 4m12s
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 52s
smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1

smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 15m
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 12m

smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1
```

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

patched version of main (to force error)

### Kubernetes Version & Environment

_No response_

### Host OS & Architecture

_No response_

### Relevant Logs and Diagnostic Output

```shell

```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。