Improve auto-healing of "insufficient amount of resource" errors
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
When this error happens, it is highly likely that there are hanging containers although they are marked as terminated in our side. Currently we are running a script to clean up such containers manually, but it would be better to automate this process as follows, by inserting an extra call of `sync_kernels()` in background:
```mermaid
sequenceDiagram
Client-->>+Manager: Create session
Manager-->>+Agent: create_kernels()
Agent-->>-Manager: InsufficientResourceError
Manager-->>+Agent: sync_kernels()
Manager-->>-Client: Report failure
Client-->>+Manager: Create session again
Manager-->>+Agent: create_kernels()
Agent-->>-Manager: should succeed now
Manager-->>-Client: Report success
```
It reports the failure upon the first encounter of `InsufficientResourceError`, but automatically triggers the synchronization operation via agent RPC. Next time the client makes another session creation requests again and if the manager assigns the same agent for it, it should succeed.
This design assumes that it is inevitable to have deviation between the actual container states and our agent's tracking database. But this will improve the user experience by auto-healing the situation.
The advantages of inserting an extra `sync_kernels()` RPC call are:
- No need to sync all agents manually but only agents who reported the insufficient resource error
- No need to configure additional cron scripts
JIRA Issue: BA-123
Contributor guide
Assessment
This issue has not been assessed yet.