agent-substrate / agent-substrate/substrate

[Bug]: Actor remains in resuming state on runsc error

Đang mở
#1,518 0 bình luận 0 reaction 1 người được giao Được @juli4n nhận Xem trên GitHub
area/api-machinery kind/bug
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

### What happened?

When launching an actor and runsc fails (e.g. `grpc ResumeActor: rpc error: code = Internal desc = while restoring "glutton" application container: while running `runsc restore`: exit status 128`), the actor remains stuck in resuming and assigned to a worker, removing that worker from the pool of available workers.

### Expected Behavior

Would expect the actor to go into crashed state or similar and the worker to be freed.

### Steps to Reproduce

You can use this patch to force a runsc error whenever resuming an actor:

https://github.com/maxsmythe/substrate/commit/6e27c956e326e66b08afa54a3d186fa8dca993bd

shell log of behavior in action:

```
smythe@smythe63:~/dev/substrate$ kubectl ate create actor probe-2 -a benchmark-workloads --template-ref sleep
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_SUSPENDED 1 0s
smythe@smythe63:~/dev/substrate$ kubectl ate resume actor probe-2 -a benchmark-workloads
Error: failed to resume actor: rpc error: code = Internal desc = while restoring pause container: while running `runsc restore`: exit status 128
smythe@smythe63:~/dev/substrate$ kubectl ate get actor probe-2 -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 30s
smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 4m12s
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 52s
smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1

smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 15m
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 12m

smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1
```

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

patched version of main (to force error)

### Kubernetes Version & Environment

_No response_

### Host OS & Architecture

_No response_

### Relevant Logs and Diagnostic Output

```shell

```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.