We are seeing a log of CHANNEL_ERROR - expected 'channel.open' 504

Hi,

We are seeing a lot of Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504 in an production environment.

What could be the cause?

amqproxy-1  | 2025-10-16T17:24:19.643850Z  INFO amq_proxy.channel_pool[remote_address: "redacted:45872"] Adding upstream connection
amqproxy-1  | 2025-10-16T17:24:36.773968Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:36:48.494735Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:38:22.619527Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:47:07.802517Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:48:09.282512Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:48:31.041006Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:08:12.175095Z  INFO amq_proxy.channel_pool[remote_address: "redacted:46308"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:11:08.115664Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:17:25.878805Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:18:13.484713Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:41:09.148686Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:42:30.450506Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:42:30.662506Z  INFO amq_proxy.channel_pool[remote_address: "redacted:60778"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:45:22.076712Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:47:09.479162Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:47:09.875097Z  INFO amq_proxy.channel_pool[remote_address: "redacted:34560"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:53:07.727178Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:53:46.052101Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:58:16.656650Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T19:01:13.753575Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T19:03:32.973562Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504

No source file or test is named. Start by tracing the amq_proxy.upstream and amq_proxy.channel_pool paths associated with the reported log entries, then reproduce or correlate the channel.open failure with the upstream connection lifecycle. Done means identifying a supported root cause and documenting the conditions and evidence needed to confirm it.