lookit / lookit/lookit-api

production-web pod out-of-memory errors and restarts

Open
#1,248 2 comments 0 reactions 1 assignee Claimed by @okaycj View on GitHub
bg-priority
Dominant language
Python
Stars
12
Forks
21
Avg merge
5d 19h
Merged PRs (30d)
5

Description

**Description**

We are experiencing problems with the production-web deployment getting its pods killed due to out-of-memory errors. We upgraded the node size from 8 to 16 GB but experienced another out-of-memory pod error/restart _after_ the memory upgrade (Aug 2, 2023, around 1:30 Eastern). This suggests that there may be a memory leak in the codebase, or processes/queries that are highly memory intensive and should be optimized.

**How to reproduce**

At this point we're not sure what is causing the production-web pods to run out of memory or how to reproduce the sharp spike in memory usage.

**Expected behavior**

Ideally the pods should not be running out of memory as this creates a poor user experience: temporary site outages, and probably contributing to the intermittent very slow page load times. Any memory leaks should be fixed, and any suboptimal processes/queries should be optimized to prevent this error.

**Screenshots**

![image](https://github.com/lookit/lookit-api/assets/9041788/fc7686b7-ba0a-47c0-919d-5158189e8f0a)
![image010](https://github.com/lookit/lookit-api/assets/9041788/c43996c0-b99c-4b78-ad64-9c2c33b9d6f1)
![Screenshot 2023-08-03 at 10 58 22 AM](https://github.com/lookit/lookit-api/assets/9041788/5d3fdb07-55ab-44d8-9787-7cfdc1edd881)

**Additional context**

Some ideas and resources:
- Python memory profiler tool: https://github.com/xrmx/pyuwsgimemhog
- Related SO question: https://stackoverflow.com/questions/25090573/django-memory-leak-possible-causes
- We could see if the memory usage is related to specific SQL queries by cross-referencing the query timestamps with the pod memory spikes and outages. We also may be able to gather some metrics from the database regarding the slowest query response times to see if there are any major outliers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.