bug: request response protocol outbound failure, UnexpectedEof

Summary

We have a server that is connected to 1000s of peers, and is using request-response protocol to send messages to them.

We seem to be getting lots of the following errors from request-response protocol OutboundFailure event with the following error:

Io(Custom { kind: UnexpectedEof, error: Eof { name: "array", expect: Small(1) } })

At high loads, more than half of our messages get this error.

Max concurrent streams number is 20000 as given to the behaviour config.

Any ideas what may be the cause of this?

Expected behavior

Should not get EoF errors as noted in summary section.

Actual behavior

Causes EoF errors as noted in summary section.

Relevant log output

[2025-05-27T11:33:15Z ERROR dkn_rpc::node::reqres::task] Task 01970504-d4f6-7000-8ac0-299950153a46/01970505-32b4-7043-80ac-9fc3cd707371 from 16Uiu2HAmVAfDDmVSK8ScJf1yrrWiYSVPVy7QBbNcJEhLv5iXQomM failed without a response: IO error on outbound stream: Eof { name: "array", expect: Small(1) }

Possible Solution

Version

0.55.0

Would you like to work on fixing this bug?

No

Start with rust-libp2p's request-response protocol, especially OutboundFailure handling and the max concurrent streams configuration, then reproduce the UnexpectedEof under the reported high peer and stream load. Use the supplied log output as the failure signature; done means identifying the cause and adding a verified fix or regression coverage.