concurrent.futures.ThreadPoolExecutor does not free memory when shutdown
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- PR merge metrics
- PR metrics pending
Description
Bug report
Memory allocated in threads is never freed when using a ThreadPoolExecutor. The expected behavior is that when the executor is shut down, all memory allocated in its threads should be freed. The below code demonstrates this leak in that the the memory usage before allocating memory in threads is significantly less than after.
from concurrent.futures import ThreadPoolExecutor, as_completed
import resource
def process_user(x):
return bytearray(10000000)
print('Before', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
def leak_memory():
with ThreadPoolExecutor(max_workers=20) as executor:
futures = [executor.submit(process_user, i) for i in range(100)]
for future in as_completed(futures):
cur = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024
print('Step', cur)
leak_memory()
print('After', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
Your environment
Python 3.9.x/3.10.x. Mac os and debian.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the example on Python 3.9 or 3.10 under macOS or Debian, focusing on concurrent.futures.ThreadPoolExecutor shutdown and the resource.RUSAGE_SELF ru_maxrss measurement. Establish whether the reported value should decrease after shutdown, then identify the relevant implementation or tests and define the expected post-shutdown memory behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100