bcgov / bcgov/itvr

ITVR - task queue creates too many database connections

Open
#778 1 comment 0 reactions 1 assignee Claimed by @tim738745 View on GitHub
Bug Medium Tested :)
Dominant language
Python
Stars
2
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Describe the Bug**
From time to time, the task queue service may create and leave open too many database connections; this impacts the ability for the backend to connect to the database.

**Expected Behaviour**
This should not happen.

**Actual Behaviour**
This does happen.

**Implications**
The backend may not be able to connect to the database, and the task queue itself is unable to make new connections to the database.

**Steps To Reproduce**
This is hard to reproduce, because it does not happen deterministically. The task queue has 4 worker processes, and the threads in each process manages its own database connections. While django-q, the library we're using, does close db connections, it only seems to close them at the beginning of each task/job. So it could be the case that a scheduled job that opens some connections executes in process 1, and then in process 2, and then process 3. This would lead to an accumulation of connections. However, if it executes in process 1 the second time around, the connections it created the first time around would be closed. We do not determine in which process a job executes. So to address this issue, we should close db connections at the end of each task/job (see scheduled_jobs.py).

Also, the job that expires rebates runs each expiry action inside a transaction; maybe each transaction opens a new connection?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.