Unitech / Unitech/pm2

Python threading causes memory leak in pm2

Open
#4,852 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

What's going wrong?

I have done some test and realized that when using threading with pm2 it causes memory leaks where it stacks up the threading instead of dumping them. I have tested both with pm2 vs normal terminal. When using threading I could see that using normal terminal would take around 60mb~ usage. While using PM2 could stack up to be over 160MB~ with the same exact code.

What I do is that when the threading is done with its task, it should be killed with a return/or sys.exit()

How could we reproduce this issue?

This is a example I have done, I'm not sure if this could replicate it but same code should give two different RAM usage even thought its the same exact code.

EDIT:

After testing on Windows. The issue still remains. I can see that it takes around 11MB~ with PM2 while with terminal/cmd it takes around 4.5MB.

#!/usr/bin/python3
# -*- coding: utf-8 -*-

import random
import threading
import time


class Monitoring(object):

    def parseNew(self):

        ListsNumber = []

        while True:

            newtLists = random.sample(range(1, 1000), 999)

            for numbers in newtLists:
                if numbers not in ListsNumber:
                    ListsNumber.append(numbers)

                    threading.Thread(
                        target=self.threadingTest,
                        args=(numbers,)
                    ).start()
            else:
                print("sleeping")
                time.sleep(random.randint(2, 4))

    def threadingTest(self, numbers):
        print(numbers)
        return


if __name__ == '__main__':
    Monitoring().parseNew()

Supporting information

Python 3.8.2, Ubuntu 20.04.1 LTS

--- PM2 report ----------------------------------------------------------------
Date                 : Fri Sep 11 2020 15:45:24 GMT+0200 (Central European Summe                                                                                                                                                             r Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.4.0
node version         : 14.6.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : autosnkr
uid                  : 1000
gid                  : 1000
uptime               : 50586min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.4.0
node version         : 14.6.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : autosnkr
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz
cpus nb              : 12
freemem              : 1734098944
totalmem             : 16617046016
home                 : /home/autosnkr
===============================================================================

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python threading script directly and under PM2, using the reported Python 3.8.2 and PM2 4.4.0 environment where possible. Compare process and memory behavior, then inspect PM2's process lifecycle and Python execution handling; done means identifying a reproducible PM2-specific leak and confirming that the same workload no longer accumulates memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, python
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.