ITVR - task queue creates too many database connections
- Dominant language
- Python
- Stars
- 2
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the Bug**
From time to time, the task queue service may create and leave open too many database connections; this impacts the ability for the backend to connect to the database.
**Expected Behaviour**
This should not happen.
**Actual Behaviour**
This does happen.
**Implications**
The backend may not be able to connect to the database, and the task queue itself is unable to make new connections to the database.
**Steps To Reproduce**
This is hard to reproduce, because it does not happen deterministically. The task queue has 4 worker processes, and the threads in each process manages its own database connections. While django-q, the library we're using, does close db connections, it only seems to close them at the beginning of each task/job. So it could be the case that a scheduled job that opens some connections executes in process 1, and then in process 2, and then process 3. This would lead to an accumulation of connections. However, if it executes in process 1 the second time around, the connections it created the first time around would be closed. We do not determine in which process a job executes. So to address this issue, we should close db connections at the end of each task/job (see scheduled_jobs.py).
Also, the job that expires rebates runs each expiry action inside a transaction; maybe each transaction opens a new connection?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.