bird-house / bird-house/twitcher

Possible memory leak with Twitcher/Magpie

Open
#113 4 comments 0 reactions 1 assignee Claimed by @cehbrecht View on GitHub
Dominant language
Python
Stars
15
Forks
5
Avg merge
4d 14h
Merged PRs (30d)
4

Description

## Describe the bug

Twitcher is constantly taking lots of Memory.

Note below Twitcher is taking 23G of memory, while the front proxy only 30M, while having similar "NET I/O".
```
$ docker stats proxy twitcher thredds
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
fad5cec748b9 proxy 13.83% 30.29MiB / 125.8GiB 0.02% 4.29TB / 4.34TB 0B / 0B 10
4612ca2a1be0 twitcher 17.63% 23.73GiB / 125.8GiB 18.86% 4.82TB / 4.83TB 0B / 0B 51
81aaf9dcbdc8 thredds 118.74% 7.014GiB / 125.8GiB 5.58% 26.6GB / 4.25TB 0B / 0B 104
```

We have had a lot of Thredds activity lately so it's normal for the Cpu and Memory consumption to increase for Thredds since it has a caching feature.

But Twitcher saw a proportionally increase which is puzzling to me and stay like this during idle time.

Related bug on Magpie https://github.com/Ouranosinc/Magpie/issues/505

## To Reproduce

PAVICS deployed at this commit from pour production fork of birdhouse-deploy https://github.com/Ouranosinc/birdhouse-deploy/commit/76dd3c86e57d6a96ca84d2124f3b019fe278645a which triple Thredds memory.

Lots of Thedds OpenDAP access on large `.ncml` file.

## Expected behavior

Twitcher Cpu and Memory would increase during Thredds transfer/activity but should go down during idle time, just like the `proxy` container.

## Context

- Instance: https://pavics.ouranos.ca/twitcher/
- Version: `[{"name": "Twitcher", "version": "0.6.2", "type": "application"}, {"name": "magpie.adapter.MagpieAdapter", "version": "3.19.1", "type": "adapter"}]`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.