production-web pod out-of-memory errors and restarts
- Dominant language
- Python
- Stars
- 12
- Forks
- 21
- Avg merge
- 5d 19h
- Merged PRs (30d)
- 5
Description
**Description**
We are experiencing problems with the production-web deployment getting its pods killed due to out-of-memory errors. We upgraded the node size from 8 to 16 GB but experienced another out-of-memory pod error/restart _after_ the memory upgrade (Aug 2, 2023, around 1:30 Eastern). This suggests that there may be a memory leak in the codebase, or processes/queries that are highly memory intensive and should be optimized.
**How to reproduce**
At this point we're not sure what is causing the production-web pods to run out of memory or how to reproduce the sharp spike in memory usage.
**Expected behavior**
Ideally the pods should not be running out of memory as this creates a poor user experience: temporary site outages, and probably contributing to the intermittent very slow page load times. Any memory leaks should be fixed, and any suboptimal processes/queries should be optimized to prevent this error.
**Screenshots**



**Additional context**
Some ideas and resources:
- Python memory profiler tool: https://github.com/xrmx/pyuwsgimemhog
- Related SO question: https://stackoverflow.com/questions/25090573/django-memory-leak-possible-causes
- We could see if the memory usage is related to specific SQL queries by cross-referencing the query timestamps with the pod memory spikes and outages. We also may be able to gather some metrics from the database regarding the slowest query response times to see if there are any major outliers.
Contributor guide
Assessment
This issue has not been assessed yet.