pyronear / pyronear/pyro-engine

Avoid host crash due to docker excessive resources use

Open
#193 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted question
Dominant language
Python
Stars
19
Forks
8
PR merge metrics
No merged PRs in 30d

Description

According to docker documentation :

By default, a container has no resource constraints and can use as much of a given resource as the host's kernel scheduler allows.

and this is how pyro-engine actually works.

However, we have recently encounter issues, rpi crash, probably due to a due to excessive use of multiprocessing and it has needed manual reboot. https://github.com/pyronear/pyro-engine/blob/feb54de42c050cc59797e2adbbcabc52d3e8a607/pyroengine/core.py#L4

To prevent this kind of situation from recurring, we could consider limiting usage by docker Memory, CPUs, and GPUs.

I'd like to know the best practices on this subject (which parameters to play with ... hard or soft limits for memory..... if the rpi has 4 CPUs, is relevant to setup a limit to 3.8 .. etc ), which limitations to implement, and whether there are any constraints to implementing this type of docker limitation.

I would tend to think that we want to minimize the limits imposed on the docker service, but make sure we avoid a complete host crash

Thank you in advance for your suggestions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pyroengine/core.py at the linked line and review Docker's resource constraints documentation. Determine which CPU, memory, and GPU limits are appropriate for the Raspberry Pi deployment and identify implementation constraints; done requires agreed limits and a documented implementation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, raspberry-pi
Domain
devops, embedded-iot, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.