Create more performant plugin so that iotile-gateway can support high volume broadcast traffic
- Dominant language
- Python
- Stars
- 14
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
The `iotile-gateway` code has been identified to not support a high enough rate of broadcast traffic to work in situations where there are many PODs (80+) that are all broadcasting data at a high advertising rate (50 ms ble advertising interval). The tested scenario was 1 bled112 dongle connected to an `iotile-gateway` instance with one `websockets` device server. There was a single client connected to the `iotile-gateway` using a `websockets` device adapter and pushing each received broadcast packet to a remote server over mqtt.
It was observed that the `websockets` connection could not keep pushing the broadcast traffic fast enough and was falling behind, with the broadcasts queuing up for transmit on the `iotile-gateway` process until eventually it ran out of memory. The test was run on a Raspberry Pi 3b+ and the `iotile-gateway` process pegged one core of the CPU at 100% indicating we were somehow CPU bound in the transfer process.
### Reproduction Steps
1. Create a running `iotile-gateway` instance with a websockets gateway agent and a bled112 device adapter. Attach the bled112 device adapter to a mock bled112 device that generates high volume advertisement traffic. On linux you can use `socat` to create a dual-ended serial port using the instructions [here](https://stackoverflow.com/a/19733677/9739119).
**Suitable Config File:**
```json
{
"adapters": [
{
"name": "bled112",
"port": "PATH/TO/VIRTUAL/COM/PORT"
}
],
"servers": [
{
"name": "websockets"
}
]
}
```
**Command:**
```
> iotile-gateway --config ABOVE_CONFIG.json -vvvvv
```
2. Vary the advertisement throughput through the fake bled112 increasing it until you hit steady state where you get back pressure from the `write()` indicating the `iotile-gateway` can't keep up. Note the advertisement throughput.
3. Profile the `iotile-gateway` and `websockets` code looking for bottlenecks in the processing
### References
- #837 contains initial debugging information from when we first saw this issue.
- API documentation on the BLED112 dongle showing how it responds to serial commands is [here](https://www.silabs.com/documents/public/reference-manuals/Bluetooth_Smart_Software-BLE-1.7-API-RM.PDF). Free account registration may be necessary for download.
### Tasks
- [x] #863 Create standalone mock bled112 class for sending high volume advertisements.
- [x] #864 Determine where bottlenecks are in `iotile-gateway` or `websockets` code for handling high volume broadcasts.
Contributor guide
Assessment
This issue has not been assessed yet.