Flashing errors with recent Windows update

A recent Windows 10 and Windows 11 update has started triggering checksum and time our errors on DAPLink.

This has been reported by micro:bit & Calliope users, and we have been able to replicate in Windows 10 and 11 when the OS is kept up-to-date. We haven't tried Windows 8.1, but that has reached end of life last January.

## Triggering Windows Update

The cumulative updates have been found and installed using this Microsoft catalogue:
https://www.catalog.update.microsoft.com/Search.aspx?q=Cumulative%20Update%20Windows%2011%2022H2%20x64

### Windows 11 22H2

I went through installing and uninstalling cumulative updates, and in my findings the problem is triggered when installing [2023-02 Cumulative Update Preview for Windows 11 Version 22H2 for x64-based Systems (**KB5022913**)](https://support.microsoft.com/en-gb/topic/february-28-2023-kb5022913-os-build-22621-1344-preview-3e38c0d9-924d-4f3f-b0b6-3bd49b2657b9) from the 28th of February, which updates Windows 11 22H2 to OS Build **22621.1344**.

The previous cumulative update KB5022845 from the 14th of Feb (OS Build 22621.1265) doesn't trigger this issue.

### Windows 11 21H2

The Microsoft update catalog doesn't show any updates for Win 11 1 21H2 since November 2022, so I won't bother to test this Windows version.

### Windows 10 22H2

The issue was triggered for me using [2023-03 Cumulative Update Preview for Windows 10 Version 22H2 for x64-based Systems (KB5023773)](https://support.microsoft.com/en-us/topic/march-21-2023-kb5023773-os-builds-19042-2788-19044-2788-and-19045-2788-preview-5850ac11-dd43-4550-89ec-9e63353fef23) from the 21st of March, which updates the OS to build **1904x.2788**.

The previous cumulative update KB5023696 from the 14th of March (OS Build 1904x.2728) doesn't trigger this issue.

### Windows 10 21H2

We've also been able to replicate this issue in Win 10 21H2, and Microsoft is still releasing updates for this OS version, so it makes sense that we could identify a specific cumulative update to introduce this issue.
I probably won't be looking into this one, nor Win 10 20H2 as it's unlikely to provide any additional useful information.

## Failure modes

> **Note**
> It's worth mentioning that the micro:bit V2 port contains an additional feature where if DAPLink encounters an error, it will reflash the target with a custom small programme that scrolls the error code in the micro:bit LED matrix display.
This is relevant because in some occasions this error programme is not flashed.

We've encountered a few different ways in which errors emerge:
- The file transfer takes a little bit less time than usual, the target programme doesn't run, DAPLink MSD drive remounts with a fail.txt
- For micro:bit specifically, in this case the error programme is flashed and error code does scrolls on the display
- fail.txt errors we've seen so far:
- Checksum error (error code 21) - the most common error
- An error occurred during the transfer (error code 3)
- Timeout error (error code 4)
- Update sent was incomplete (error code 37)
- Blocks out of order (error code 6)
- The file transfer gets stuck at some point and can take a significant time before it errors (in my test environment with Win 11 2 to 3 minutes)
- Explorer shows an error transferring the file, first screenshoot below
- The target programme doesn't run, the DAPLink MSD drive remounts with one of the two assert.txt files listed below
- I'm not 100% sure, but I think possibly DAPLink is crashing in this scenario, as I am using Windows VMs and the MSD drive unmounts from the VM and mounts back in my host OS, which usually only happens when DAPLink crashes.
- In some cases the explorer error is different, as shown in screenshots 2 and 3
- I haven't been able to replicate these myself, but have gotten multiple people reporting them, so trying to get more info at the moment

The errors are not triggered on every flash, but different users have reported different error frequencies. In our internal testing some teammates measured 20% failure rate and others up to 60%. Some users have reported errors happenning on "almost every flash".

We've used micro:bit Universal Hex files for the majority of these tests, which are a bit more resilient to this issue (more info in the "Identifying the Cause" section), so other DAPLink users flashing Intel Hex files might encounter this problem more often (it's also likely that the micro:bit user that reported an error on "almost every flash" was using Intel Hex files as well).

| | |
|-----|-----|
| screenshot1 | |
| screenshot2 | |
| screenshot3 | |

```
Assert
File: ../../../source/daplink/drag-n-drop/vfs_manager.c
Line: 361
Source: Application
Hexdumps
fffffff1
20000fc0
20005e88
00000000
20005eac
00000000
00000000
1fffeb9c
```

```
Assert
File: ../../../source/daplink/drag-n-drop/vfs_manager.c
Line: 361
Source: Application
Hexdumps
fffffff1
20000fc0
20005e88
00000000
20005eac
00000000
00000000
1fffeb9c
```

## Identifying the Cause

I’ve collected a couple of RTT logs from DAPLink with additional debug prints to track how the OS writes the file blocks to disk, and peaking at the actual data. While it’s still a bit early (I need more time to capture more data and analyse it), initial findings point at the problem being caused by file blocks being sent out of order by the OS.

In previous Windows versions, the file blocks are sent in order, but after the listed Windows updates are installed it looks like some file blocks are first sent as zeros, and then later down the file transfer the blocks are sent again with the real file data.

For example:
- OS sends file blocks 0 to 512 correctly
- Blocks 512 to 1024 are sent as zeros
- Blocks 1024 to 1536 are sent correctly
- Real data for blocks 512 to 1024 are sent at this point

And this can happen more than once on the same file transfer.

However, not every file transfer sends files out of order, some are sent in order and it all works fine.

The check sum errors are encountered when the OS sends a block filled with zeros and DAPLink tries to calculate the checksum of an Intel Hex record. I still need to capture a better log for timeout errors, but I believe those are usually triggered when out of order blocks are ignored by DAPLink and then when the OS has finished sending the file, then DAPLink waits for more data to arrive (as the ignored blocks are not counted when measuring how much file data was transferred) until it eventually times out.

For the micro:bit specifically we use [Universal Hex](https://tech.microbit.org/software/hex-format/#universal-hex-files) files, a superset of the Intel Hex format, which contains data for micro:bit V1 and micro:bit V2 in the same file. In file transfers where the out-of-order blocks correspond only for a section of the Universal Hex file that is not relevant the target MCU being flashed, the flash can still be successful. So while I haven't yet compared failure rates of Intel vs Universal Hex, it's very likely Intel Hex (and bin) files fail more frequently.

A checksum error log and Universal Hex file can be found here:
- [checksum-error-log.zip](https://github.com/ARMmbed/DAPLink/files/11352547/checksum-error-log.zip)

(Also note that because there is a lot of log data captured, data is sometimes dropped, so it might look like some blocks are not being sent, but we can look at the variables tracking the file size transferred to confirm that data has been processed, it's just that the RTT buffer was likely full).

## Workarounds

Using `robocopy` with the `/z` flag, for restartable mode, seems to be work so far.

For example, with the terminal at the path where your `file.hex` is located, and assuming DAPLink is mounted as drive `E:\`:
```
robocopy /z . E:\ file.hex
```

Also, WebUSB flashing works, so for Intel Hex and bin files this demo from DAPJs can still flash the boards:
https://armmbed.github.io/dapjs/examples/daplink-flash/web.html

For micro:bit Universal Hex files, with online WebUSB tool will work too:
https://microbit.org/tools/webusb-hex-flashing/

Start with source/daplink/drag-n-drop/vfs_manager.c around line 361, then reproduce the failures using the listed Windows updates and the supplied RTT log and Universal Hex file. Compare ordered and out-of-order file-block handling across Intel Hex, binary, and Universal Hex transfers. Done should include an agreed fix for the affected transfer behavior and evidence that the reported checksum, timeout, and assert failures no longer occur.

Written by the indexing model from the issue text.