ElementsProject / ElementsProject/lightning
Zombie connection after lightningd ignores `connectd_peer_spoke`
- Dominant language
- C
- Stars
- 3.1k
- Forks
- 1k
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 13
Description
When connectd reads a peer message whose `channel_id` has no attached subd, it creates a subd with `conn == NULL` and sends `connectd_peer_spoke` to lightningd, asking it to start up a subdaemon and respond with `connectd_peer_connect_subd` and the file descriptor that should be assigned to `conn`.
While connectd is waiting for lightningd's response, it **stops reading all messages from the peer**:
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/connectd/multiplex.c#L1506-L1538
The `io_wait` at the end is only released when either lightningd responds, or when CLN needs to send a message to that peer (e.g., gossip flush). If neither happens, the connection becomes a zombie and all peer messages are ignored until eventually the ping timeout expires and the connection is dropped.
There are currently six situations where lightningd's `handle_peer_spoke` never responds:
1. The peer sends an `error` for its first message (e.g., data loss recovery).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2067-L2072
2. channeld dies from a bug or protocol violation, and the peer sends another message for that channel before lightningd recognizes the channeld died. An alternative (benign) way to trigger this is when `lightningd` has already created the requested subdaemon but `connectd` hasn't processed it yet.
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2074-L2081
3. The peer attempts a `channel_reestablish` while the node is shutting down.
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2083-L2097
4. The peer sends a message after channeld was killed without a status message (e.g., OOM).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2100-L2109
5. openingd fails to spawn (e.g., fd limit reached).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2143-L2145
6. dualopend fails to spawn (e.g., fd limit reached).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2163-L2165
### Suggested fix
Add a new message `connectd_peer_no_subd` that lightningd can respond with when `handle_peer_spoke` fails in the above situations. Then connectd knows to free the matching subd and continue reading from the connection.
### Discovery
This bug was discovered while fuzzing CLN with [smite](https://github.com/morehouse/smite/). Smite would send a `channel_ready` message with an incorrect `channel_id`, causing channeld to exit. Then smite would send another channel message, which would trigger the Situation 2 race condition and zombify the connection.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked handling in connectd/multiplex.c and lightningd/peer_control.c, especially the six paths where handle_peer_spoke does not respond. Trace the proposed connectd_peer_no_subd message through both components; done means failed subdaemon startup or lookup frees the matching subd and connectd resumes reading peer messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100