MAVLink over DroneCAN: Systematic loss of specific MAVlink messages

## Bug report

mLRS has DroneCAN support for some months now, and respective hardware by a manufacturer and ArduPilot partner is in the pipeline, but we are observing an issue with the MAVLink via DroneCAN telemetry stream. While I cannot prove that the issue is not on mLRS side, I believe strong evidence points to the possibility there is an issue in ArduPilot here.

The issue manifests itself as a packet loss reported by MissionPlanner in its stats window.

**Issue details**

The system we consider is ArduPlane master (of two days ago) running on an H743 flight controller, with a mLRS receiver connected via DroneCAN and both devices being configured for DroneCAN RC and MAVLink via DroneCAN (i.e. the DroneCAN serial feature is used).

The stream rates are set as follows:
SRx_ADBS = 5
SRx_EXT_STAT = 2
SRx_EXTRA1 = 4
SRx_EXTRA2 = 4
SRx_EXTRA3 = 2
SRx_PARAMS = 10
SRx_PPSOITION = 2
SRx_RAW_CNTRL = 1
SRx_RAW_SENSE = 2
SRx_RC_CHAN = 2

This gives a telemetry data rate of ca 1.7 kBytes/sec or ca 56 mavlink packets/sec. This is well below the capacity limit of the mLRS link in the used configuration, that is no flow control or loss over the link is present.

The telemetry stream contains "clusters" of the three MAVLink messages ATTITUDE (id = 30), VFR-HUD (id = 74), AOA_SSA (id = 11020), which always follow straight after each other in this sequence. According to the stream rate settings these clusters should appear 4 times per sec, every ca 250 ms a cluster.

I have added debug code to the mLRS receiver which prints out the received MAVLink messages, with time stamp, and further debug info, to see what the mLRS receiver "sees". This does not rule out the possibility that the mLRS CAN driver regularly misses some tunnel.Targetted frames (I sadly could not find a way how to disproove this), but the evidence collected points towrads an issue in AP.

The following is known (as experimental fact) about this packet loss (in steady state operation, i.e., no GCS, no param uploads or such):

- when a normal serial connection is used between the mLRS receiver and the flight controller, no packet loss occurs. However, a detailed log of the MAVLink packets shows some irregulurities, which will be commented on further below.

- for nearly all of the time it is a loss of exactly 3 packets per sec, observable in the stats window of MissionPlanner. Meaning there is jump in the messages sequence number by 3. If it is not 3, then it often is a multiple of 3, like 6 etc.pp, sometimes also 0.

- the message loss is ALWAYS associated to a loss of a (30,74,11020) cluster. I never did observe any other packet loss. It can happen that two such (30,74,11020) clusters are lost per second, but most of the time it is one cluster loss per second.

- the expected four (30,74,11020) clusters per sec do not come at intervals of ca 250 ms, but appear in pairs of two such clusters, with no other messages in between, and with a time gap of 100 ms or less. This is so ALSO for then a normal serial connection is used between receiver and flight controller!

- the message loss is ALWAYS a loss of the second cluster in such a pair of clusters. That is, instead of the sequence 30, 74, 11020, 30, 74, 11020 one only receives 30, 74, 11020, and the jump in the messages sequence number happens AFTER the 11020 message.

- the issue is not due to any buffer overflow or alike in the mLRS code. I have added extensive debug code for any buffer/fifo which is in the game here, and they all are less than half filled.

- I also added extensive debug code to detect CAN farme errors or losses as much as I could, and none triggers. (not a proove for that the issue cannot be on the mLRS side, just stating)

- removing/outcommenting this line https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_DroneCAN/AP_DroneCAN.cpp#L526 and increasing the timeout to 100 in this line https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_DroneCAN/AP_DroneCAN_serial.cpp#L59 makes the issue to almost go away. The cluster loss can still happen in certain conditions, but much much less so.

Given that it are always exactly these three messages which are lost and always the 2nd in the pair, makes it hard to see how it should be coming from issues in the mLRS CAN driver (which doesn't know about the MAVLink content, and hardly would cut out systematicaly exactly the corresponding frames). Comes in in addition the other evidence.

--------------------------

I'd like to add two logs generated with the debig code in the mLRS receiver. They show

timestamp in ms, message id, message seq number, the symbol ERR if a jump in the seq number is detected

Shown is the snapshot for one sec, from a HEARTBEAT message to the next HEARBEAT message. Comments are added with '<-' to point to specific places.

Log with MAVLink via DroneCAN (and an example of two cluster losses)

--
m t 08168 id 00000 seq 00168
m t 08203 id 00030 seq 00169
m t 08203 id 00074 seq 00170
m t 08221 id 11020 seq 00171
m t 08503 id 00163 seq 00175 ERR <- packet loss of 3 packets, expected would another 30,74,11020 cluster
m t 08503 id 00033 seq 00176
m t 08503 id 00001 seq 00177
m t 08505 id 00125 seq 00178
m t 08505 id 00152 seq 00179
m t 08505 id 00062 seq 00180
m t 08505 id 00042 seq 00181
m t 08507 id 00036 seq 00182
m t 08508 id 00065 seq 00183
m t 08538 id 00027 seq 00184
m t 08538 id 00116 seq 00185
m t 08538 id 00029 seq 00186
m t 08591 id 00024 seq 00187
m t 08591 id 00002 seq 00188
m t 08591 id 00168 seq 00189
m t 08591 id 00136 seq 00190
m t 08592 id 00193 seq 00191
m t 08644 id 00241 seq 00192
m t 08644 id 00087 seq 00193
m t 08644 id 00147 seq 00194
m t 08645 id 11039 seq 00195
m t 08702 id 00030 seq 00196
m t 08702 id 00074 seq 00197
m t 08702 id 11020 seq 00198.
m t 09002 id 00163 seq 00202 ERR <- packet loss of 3 packets, expected would another 30,74,11020 cluster
m t 09003 id 00033 seq 00203
m t 09003 id 00001 seq 00204
m t 09005 id 00125 seq 00205
m t 09005 id 00152 seq 00206
m t 09005 id 00062 seq 00207
m t 09005 id 00042 seq 00208
m t 09005 id 00036 seq 00209
m t 09008 id 00065 seq 00210
m t 09015 id 00027 seq 00211
m t 09015 id 00116 seq 00212
m t 09015 id 00029 seq 00213
m t 09068 id 00024 seq 00214
m t 09068 id 00002 seq 00215
m t 09068 id 00168 seq 00216
m t 09068 id 00136 seq 00217
m t 09068 id 00193 seq 00218
m t 09121 id 00241 seq 00219
m t 09121 id 00087 seq 00220
m t 09121 id 00147 seq 00221
m t 09121 id 11039 seq 00222
--

Log for the case of a normal serial connection between receiver and flight controller. I did similar logs by directly sniffing on the serial lines with a usb-ttl adapter and a pymavlink base python code. The two approaches yield consistent results.

--
m t 30292 id 00000 seq 00123
m t 30292 id 00030 seq 00124
m t 30292 id 00074 seq 00125
m t 30345 id 11020 seq 00126
m t 30404 id 00030 seq 00127 <- 2nd 30,74,11020 cluster in a pair of clusters, it is not ca 250 ms later, but just ca 60 ms here
m t 30409 id 00074 seq 00128
m t 30412 id 11020 seq 00129
m t 30604 id 00163 seq 00130
m t 30611 id 00033 seq 00131
m t 30616 id 00001 seq 00132
m t 30619 id 00125 seq 00133
m t 30622 id 00152 seq 00134
m t 30627 id 00062 seq 00135
m t 30630 id 00042 seq 00136
m t 30634 id 00036 seq 00137
m t 30643 id 00065 seq 00138
m t 30650 id 00027 seq 00139
m t 30662 id 00116 seq 00140
m t 30662 id 00029 seq 00141
m t 30667 id 00024 seq 00142
m t 30671 id 00002 seq 00143
m t 30715 id 00168 seq 00144
m t 30715 id 00136 seq 00145
m t 30716 id 00193 seq 00146
m t 30716 id 00241 seq 00147
m t 30716 id 00087 seq 00148
m t 30768 id 00147 seq 00149
m t 30768 id 11039 seq 00150
m t 30768 id 00030 seq 00151
m t 30821 id 00074 seq 00152
m t 30821 id 11020 seq 00153
m t 30903 id 00030 seq 00154 <- 2nd 30,74,11020 cluster in a pair of clusters, it is not ca 250 ms later, but just ca 80 ms here
m t 30908 id 00074 seq 00155
m t 30911 id 11020 seq 00156
m t 31104 id 00163 seq 00157
m t 31111 id 00033 seq 00158
m t 31116 id 00001 seq 00159
m t 31118 id 00125 seq 00160
m t 31121 id 00152 seq 00161
m t 31126 id 00062 seq 00162
m t 31129 id 00042 seq 00163
m t 31133 id 00036 seq 00164
m t 31142 id 00065 seq 00165
m t 31150 id 00027 seq 00166
m t 31156 id 00116 seq 00167
m t 31160 id 00029 seq 00168
m t 31192 id 00024 seq 00169
m t 31192 id 00002 seq 00170
m t 31192 id 00168 seq 00171
m t 31192 id 00136 seq 00172
m t 31192 id 00193 seq 00173
m t 31245 id 00241 seq 00174
m t 31245 id 00087 seq 00175
m t 31245 id 00147 seq 00176
m t 31245 id 11039 seq 00177
--

Start by inspecting libraries/AP_DroneCAN/AP_DroneCAN.cpp at line 526 and libraries/AP_DroneCAN/AP_DroneCAN_serial.cpp at line 59, then reproduce the reported MAVLink cluster loss over DroneCAN while comparing the normal serial connection. Done means identifying and correcting the source of the systematic ATTITUDE, VFR-HUD, and AOA_SSA losses, with the reported telemetry stream no longer dropping those clusters.

Written by the indexing model from the issue text.