iotile / iotile/coretools

Create more performant plugin so that iotile-gateway can support high volume broadcast traffic

Open
#865 5 comments 0 reactions 2 assignees Claimed by @timothyfroehlich View on GitHub
major-feature
Dominant language
Python
Stars
14
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Overview

The `iotile-gateway` code has been identified to not support a high enough rate of broadcast traffic to work in situations where there are many PODs (80+) that are all broadcasting data at a high advertising rate (50 ms ble advertising interval). The tested scenario was 1 bled112 dongle connected to an `iotile-gateway` instance with one `websockets` device server. There was a single client connected to the `iotile-gateway` using a `websockets` device adapter and pushing each received broadcast packet to a remote server over mqtt.

It was observed that the `websockets` connection could not keep pushing the broadcast traffic fast enough and was falling behind, with the broadcasts queuing up for transmit on the `iotile-gateway` process until eventually it ran out of memory. The test was run on a Raspberry Pi 3b+ and the `iotile-gateway` process pegged one core of the CPU at 100% indicating we were somehow CPU bound in the transfer process.

### Reproduction Steps

1. Create a running `iotile-gateway` instance with a websockets gateway agent and a bled112 device adapter. Attach the bled112 device adapter to a mock bled112 device that generates high volume advertisement traffic. On linux you can use `socat` to create a dual-ended serial port using the instructions [here](https://stackoverflow.com/a/19733677/9739119).

**Suitable Config File:**
```json
{
"adapters": [
{
"name": "bled112",
"port": "PATH/TO/VIRTUAL/COM/PORT"
}
],

"servers": [
{
"name": "websockets"
}
]
}
```

**Command:**
```
> iotile-gateway --config ABOVE_CONFIG.json -vvvvv
```

2. Vary the advertisement throughput through the fake bled112 increasing it until you hit steady state where you get back pressure from the `write()` indicating the `iotile-gateway` can't keep up. Note the advertisement throughput.

3. Profile the `iotile-gateway` and `websockets` code looking for bottlenecks in the processing

### References

- #837 contains initial debugging information from when we first saw this issue.
- API documentation on the BLED112 dongle showing how it responds to serial commands is [here](https://www.silabs.com/documents/public/reference-manuals/Bluetooth_Smart_Software-BLE-1.7-API-RM.PDF). Free account registration may be necessary for download.

### Tasks

- [x] #863 Create standalone mock bled112 class for sending high volume advertisements.
- [x] #864 Determine where bottlenecks are in `iotile-gateway` or `websockets` code for handling high volume broadcasts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.