@rustyrussell: Protocol-triggered unilateral closure due to deadline hit on HTLC with our `revoke_and_ack` not known to have been received

@rustyrussell: What follows is a detailed analysis of the failure of a very active channel due to the peer's neglect to remove an HTLC before its deadline. The HTLC in question had reached the `SENT_ADD_ACK_REVOCATION` state just prior to the peer connection being lost, and we didn't receive any messages from the peer that would indicate that it had received our `revoke_and_ack` (or any message we sent subsequently), so it is possible that it did not. The peer is known to have been running LND 0.18.5.

> **Rusty Russell:** This is *supposed* to be handled by the counters in the reestablish message: you can tell the peer didn't get your revoke_and_ack and thus retransmit.

I am sure that much of the following discussion will be superfluous, but I want to give context to show that the channel was working correctly and to highlight the overlapped nature of the HTLCs, which hopefully may lend a clue as to why the peer evidently didn't believe it needed to remove one of our HTLCs before its deadline.

## Overview

1. HTLC 35768: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
2. HTLC 35768: `RCVD_REMOVE_HTLC`⇒`RCVD_REMOVE_COMMIT`
3. HTLC 35768: `RCVD_REMOVE_COMMIT`⇒`SENT_REMOVE_REVOCATION`
4. HTLC 35768: `SENT_REMOVE_REVOCATION`⇒`SENT_REMOVE_ACK_COMMIT`
5. HTLC 35769: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
6. HTLC 35771: **(new)** `SENT_ADD_HTLC`
7. HTLC 35769: `RCVD_REMOVE_HTLC`⇒`RCVD_REMOVE_COMMIT`
8. HTLC 35769: `RCVD_REMOVE_COMMIT`⇒`SENT_REMOVE_REVOCATION`
9. HTLC 35772: **(new)** `SENT_ADD_HTLC`
10. HTLC 35768: `SENT_REMOVE_ACK_COMMIT`⇒`RCVD_REMOVE_ACK_REVOCATION`
11. HTLC 35769: `SENT_REMOVE_REVOCATION`⇒`SENT_REMOVE_ACK_COMMIT`
HTLC 35771: `SENT_ADD_HTLC`⇒`SENT_ADD_COMMIT`
HTLC 35772: `SENT_ADD_HTLC`⇒`SENT_ADD_COMMIT`
12. HTLC 35769: `SENT_REMOVE_ACK_COMMIT`⇒`RCVD_REMOVE_ACK_REVOCATION`
13. HTLC 35771: `RCVD_ADD_REVOCATION`⇒`RCVD_ADD_ACK_COMMIT`
HTLC 35772: `RCVD_ADD_REVOCATION`⇒`RCVD_ADD_ACK_COMMIT`
14. HTLC 35771: `RCVD_ADD_ACK_COMMIT`⇒`SENT_ADD_ACK_REVOCATION`
HTLC 35772: `RCVD_ADD_ACK_COMMIT`⇒`SENT_ADD_ACK_REVOCATION`
15. HTLC 35770: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
16. HTLC 35773: **(new)** `SENT_ADD_HTLC`
17. HTLC 35773: `SENT_ADD_HTLC`⇒`SENT_ADD_COMMIT`
18. HTLC 35770: `RCVD_REMOVE_HTLC`⇒`RCVD_REMOVE_COMMIT`
19. HTLC 35770: `RCVD_REMOVE_COMMIT`⇒`SENT_REMOVE_REVOCATION`
20. HTLC 35771: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
21. HTLC 35773: `SENT_ADD_COMMIT`⇒`RCVD_ADD_REVOCATION`
22. HTLC 35770: `SENT_REMOVE_REVOCATION`⇒`SENT_REMOVE_ACK_COMMIT`
23. HTLC 35771: `RCVD_REMOVE_HTLC`⇒`RCVD_REMOVE_COMMIT`
HTLC 35773: `RCVD_ADD_REVOCATION`⇒`RCVD_ADD_ACK_COMMIT`
24. HTLC 35771: `RCVD_REMOVE_COMMIT`⇒`SENT_REMOVE_REVOCATION`
HTLC 35773: `RCVD_ADD_ACK_COMMIT`⇒`SENT_ADD_ACK_REVOCATION`
25. HTLC 35770: `SENT_REMOVE_ACK_COMMIT`⇒`RCVD_REMOVE_ACK_REVOCATION`
26. HTLC 35772: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
27. HTLC 35771: `SENT_REMOVE_REVOCATION`⇒`SENT_REMOVE_ACK_COMMIT`
28. HTLC 35774: **(new)** `SENT_ADD_HTLC`
29. HTLC 35772: `RCVD_REMOVE_HTLC`⇒`SENT_ADD_ACK_REVOCATION` **(reverted)**
HTLC 35774: **(dropped)**
30. HTLC 35774: **(new)** `SENT_ADD_HTLC`
31. HTLC 35774: **(dropped)**
32. HTLC 35774: **(new)** `SENT_ADD_HTLC`
33. HTLC 35772: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
34. HTLC 35772: `RCVD_REMOVE_HTLC`⇒`SENT_ADD_ACK_REVOCATION` **(reverted)**
HTLC 35774: **(dropped)**
35. HTLC 35772: `SENT_ADD_ACK_REVOCATION`⇒`RCVD_REMOVE_HTLC`
36. HTLC 35772: `RCVD_REMOVE_HTLC`⇒`RCVD_REMOVE_COMMIT`
37. HTLC 35772: `RCVD_REMOVE_COMMIT`⇒`SENT_REMOVE_REVOCATION`
38. HTLC 35771: `SENT_REMOVE_ACK_COMMIT`⇒`RCVD_REMOVE_ACK_REVOCATION`
39. HTLC 35772: `SENT_REMOVE_REVOCATION`⇒`SENT_REMOVE_ACK_COMMIT`
40. HTLC 35772: `SENT_REMOVE_ACK_COMMIT`⇒`RCVD_REMOVE_ACK_REVOCATION`
41. HTLC 35773: **(deadline)**

(Detailed analysis of the above to follow in comments due to GitHub's post length limit.)