ipython / ipython/ipyparallel

Static memory used by engines

Aberta
#694 2 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Jupyter Notebook
Estrelas
2.6k
Forks
1k
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

It seems like the engines use a lot of "static" memory, i.e. when we start the engines, they take up noticeable memory even when nothing has been executed. In our testing, we started 16 engines across 2 nodes, and as soon as we did, the memory usage on each node went from ~150MB to ~1.2GB, i.e. it seems like 8 engines consume ~1.05GB static memory. This is of course both the engine and nanny process, but based on `htop` , almost all of it seems to come from the engine process. ~134MB static memory usage per engine seems a bit high. Is there a way we could reduce this?

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start by reproducing the reported setup: launch 16 engines across two nodes and compare each node's baseline memory with its memory after startup, using htop as described. Separate engine and nanny process usage, then identify what accounts for the engine's static allocation. Done means reducing the idle per-engine memory footprint and verifying the change with the same measurement.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
jupyter, python
Domínio
distributed-systems, performance
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Precisa de esclarecimento
Facilidade para iniciantes
25/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.