DPIclimate / DPIclimate/broker

tswriter process does not crash when database connection is lost

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3
Forks
3
Avg merge
3m
Merged PRs (30d)
1

Description

The tswriter process kept reporting it's database connection was closed, but this did not crash the process which would have triggered docker to restart it and solve the problem.

It looks like a try/except catches the error and doesn't look closely enough to realise it should be fatal, or a new connection is required.

```
prod-tswriter-1 | 2025-09-10T09:10:27+1000|INFO |fceecfec-6084-4263-b13d-883f22282351|BaseWriter|Adding message to delivery table
prod-tswriter-1 | 2025-09-10T09:10:27+1000|INFO |BaseWriter|Processing 1 messages
prod-tswriter-1 | 2025-09-10T09:10:27+1000|INFO |fceecfec-6084-4263-b13d-883f22282351|BaseWriter|msg from table 15413, 0
prod-tswriter-1 | 2025-09-10T09:10:27+1000|INFO |fceecfec-6084-4263-b13d-883f22282351|BaseWriter|Accepted message from physical / logical device ids 426 / 381
prod-tswriter-1 | 2025-09-10T09:10:27+1000|INFO |fceecfec-6084-4263-b13d-883f22282351|BaseWriter|Wombat-B8D61A0169A0 / SCMN-lite-Dareton
prod-tswriter-1 | 2025-09-10T09:10:27+1000|ERROR |fceecfec-6084-4263-b13d-883f22282351|TimescaleDB|Error while processing message.
prod-tswriter-1 | Traceback (most recent call last):
prod-tswriter-1 | File "/home/broker/python/delivery/TimescaleDB.py", line 60, in on_message
prod-tswriter-1 | with _conn.cursor() as curs:
prod-tswriter-1 | ^^^^^^^^^^^^^^
prod-tswriter-1 | psycopg2.InterfaceError: connection already closed
prod-tswriter-1 |
prod-tswriter-1 | During handling of the above exception, another exception occurred:
prod-tswriter-1 |
prod-tswriter-1 | Traceback (most recent call last):
prod-tswriter-1 | File "/home/broker/python/delivery/TimescaleDB.py", line 65, in on_message
prod-tswriter-1 | _conn.rollback()
prod-tswriter-1 | psycopg2.InterfaceError: connection already closed
prod-tswriter-1 | 2025-09-10T09:10:27+1000|ERROR |fceecfec-6084-4263-b13d-883f22282351|BaseWriter|Message processing failed, dropping message.
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in python/delivery/TimescaleDB.py at on_message, especially the connection use and exception handling around lines 60-65 shown in the traceback. Reproduce or inspect the closed-connection path and determine whether the intended result is a fatal process exit for Docker restart or connection recovery. Done means a lost database connection no longer leaves tswriter running while repeatedly dropping messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.