ElementsProject / ElementsProject/lightning

Zombie connection after lightningd ignores `connectd_peer_spoke`

Open
#9,369 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
3.1k
Forks
1k
Avg merge
4d 10h
Merged PRs (30d)
13

Description

When connectd reads a peer message whose `channel_id` has no attached subd, it creates a subd with `conn == NULL` and sends `connectd_peer_spoke` to lightningd, asking it to start up a subdaemon and respond with `connectd_peer_connect_subd` and the file descriptor that should be assigned to `conn`.

While connectd is waiting for lightningd's response, it **stops reading all messages from the peer**:

https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/connectd/multiplex.c#L1506-L1538

The `io_wait` at the end is only released when either lightningd responds, or when CLN needs to send a message to that peer (e.g., gossip flush). If neither happens, the connection becomes a zombie and all peer messages are ignored until eventually the ping timeout expires and the connection is dropped.

There are currently six situations where lightningd's `handle_peer_spoke` never responds:

1. The peer sends an `error` for its first message (e.g., data loss recovery).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2067-L2072

2. channeld dies from a bug or protocol violation, and the peer sends another message for that channel before lightningd recognizes the channeld died. An alternative (benign) way to trigger this is when `lightningd` has already created the requested subdaemon but `connectd` hasn't processed it yet.
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2074-L2081

3. The peer attempts a `channel_reestablish` while the node is shutting down.
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2083-L2097

4. The peer sends a message after channeld was killed without a status message (e.g., OOM).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2100-L2109

5. openingd fails to spawn (e.g., fd limit reached).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2143-L2145

6. dualopend fails to spawn (e.g., fd limit reached).
https://github.com/ElementsProject/lightning/blob/ae53e8775e57ddf8debfd6fc7b8e3cb5e2c93dc6/lightningd/peer_control.c#L2163-L2165

### Suggested fix

Add a new message `connectd_peer_no_subd` that lightningd can respond with when `handle_peer_spoke` fails in the above situations. Then connectd knows to free the matching subd and continue reading from the connection.

### Discovery

This bug was discovered while fuzzing CLN with [smite](https://github.com/morehouse/smite/). Smite would send a `channel_ready` message with an incorrect `channel_id`, causing channeld to exit. Then smite would send another channel message, which would trigger the Situation 2 race condition and zombify the connection.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked handling in connectd/multiplex.c and lightningd/peer_control.c, especially the six paths where handle_peer_spoke does not respond. Trace the proposed connectd_peer_no_subd message through both components; done means failed subdaemon startup or lookup frees the matching subd and connectd resumes reading peer messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.