DPIclimate / DPIclimate/broker
Remove database access from the receiving processes
- Dominant language
- Python
- Stars
- 3
- Forks
- 3
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
At present the receiving processes use the database to find or create a physical device to associate with the received message.
The TTN receiver was implemented in two parts which shows how the others can be split such that one process can run outside a firewall and the other inside the firewall, with only outgoing connections from the second process:
```
FW
Unsafe receiver -> RMQ EX <-|- Receiver with db access -> RMQ EX -> Logical mapper -> RMQ EX -> delivery processes
```
An alternative is to publish the messages to the `physical_timeseries` exchange without physical device IDs and either have a dedicated process inside the firewall to do the create/update or have the logical mapper do it.
This would require adding a receiver identifier to the message because at present the receiver knows everything necessary to find or create the physical device and processes inside the firewall do not have this info. Perhaps the receiver module can have a find/create device function that can be used inside the firewall so the code is still associated with the receiver module.
A RabbitMQ server will need to run outside the firewall, which the receivers write their messages to, and it only accepts external connections via MQTT. MQ connections will only be accepted from the firewalled hosts.
An additional benefit of this change is that database updates can be done without disturbing the receiver processes.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests. Start by tracing the TTN receiver split, RabbitMQ exchanges, physical_timeseries messages, and logical mapper flow, then determine where device association currently occurs. Done means receiving processes no longer access the database while received messages still acquire or update physical-device associations through a firewalled process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rabbitmq
- Domain
- databases, distributed-systems, networking
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100