lablup / lablup/backend.ai

Improve auto-healing of "insufficient amount of resource" errors

Open
#2,142 0 comments 0 reactions 1 assignee Claimed by @fregataa View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

When this error happens, it is highly likely that there are hanging containers although they are marked as terminated in our side. Currently we are running a script to clean up such containers manually, but it would be better to automate this process as follows, by inserting an extra call of `sync_kernels()` in background:

```mermaid
sequenceDiagram
Client-->>+Manager: Create session
Manager-->>+Agent: create_kernels()
Agent-->>-Manager: InsufficientResourceError
Manager-->>+Agent: sync_kernels()
Manager-->>-Client: Report failure
Client-->>+Manager: Create session again
Manager-->>+Agent: create_kernels()
Agent-->>-Manager: should succeed now
Manager-->>-Client: Report success
```

It reports the failure upon the first encounter of `InsufficientResourceError`, but automatically triggers the synchronization operation via agent RPC. Next time the client makes another session creation requests again and if the manager assigns the same agent for it, it should succeed.

This design assumes that it is inevitable to have deviation between the actual container states and our agent's tracking database. But this will improve the user experience by auto-healing the situation.

The advantages of inserting an extra `sync_kernels()` RPC call are:

- No need to sync all agents manually but only agents who reported the insufficient resource error
- No need to configure additional cron scripts

JIRA Issue: BA-123

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.