celestiaorg / celestiaorg/celestia-node

v0.31.3: websocket subscriptions fail with bad handshake after RPC hardening

Open
#5,071 0 comments 0 reactions 0 assignees View on GitHub
external needs:triage
Dominant language
Go
Stars
996
Forks
1.1k
Avg merge
1d 6h
Merged PRs (30d)
34

Description

## Summary

After upgrading to `celestia-node v0.31.3`, websocket-based subscriptions started failing with `websocket: bad handshake` and clients fall back to polling.

## Environment

- Version: `v0.31.3`
- Network: observed on Mocha
- Deployment: celestia-nodes behind a TCP load balancer
- The same load balancer setup worked before upgrading
- `[RPC.RateLimit] Enabled = false`

Relevant config:

```toml
[RPC]
Address = "localhost"
Port = "36659"
SkipAuth = false
TLSEnabled = false

[RPC.RateLimit]
Enabled = false
RequestsPerSec = 100
Burst = 200
CacheSize = 8192
```

## Errors

```text
WRN DA websocket connection failed, falling back to DA polling error="failed to connect to header namespace: cannot dial address for websocket: bad handshake: websocket: bad handshake" component=main
```

A related subscription error was also observed:

```text
DA subscription failed, reconnecting error="subscribe namespace 0: blob subscribe: RPC error (-32601): method 'blob.Subscribe' not supported in this mode (no out channel support)" component=syncer
```

## Suspected area

`v0.31.3` includes RPC hardening from https://github.com/celestiaorg/celestia-node/pull/4909, including an always-on `maxConcurrentConns = 500` middleware around the RPC handler.

Because websocket upgrades hold the handler open for the lifetime of the websocket connection, this limit may effectively cap active websocket subscriptions. If saturated, new websocket upgrade requests could receive an HTTP rejection and surface client-side as `websocket: bad handshake`.

The optional per-IP rate limiter is disabled in this setup, so this does not appear to be the `[RPC.RateLimit]` path.

## Questions

1. Are websocket subscriptions expected to remain supported in `v0.31.3`?
2. Can the always-on `maxConcurrentConns = 500` reject websocket upgrades under load?
3. Should websocket connections have a separate configurable limit from normal HTTP RPC requests?
4. Should `maxConcurrentConns` be operator-configurable?
5. Can failed websocket upgrades log or expose the HTTP status/reason?

## Expected behavior

Websocket subscriptions should continue to work after upgrading to `v0.31.3`, or any new limit affecting websocket subscriptions should be documented and configurable.

## Actual behavior

Websocket handshakes fail with `websocket: bad handshake`, and clients fall back to polling.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the RPC hardening described in pull request 4909 and locate the always-on maxConcurrentConns middleware around the RPC handler. Trace the websocket upgrade path and compare it with the blob.Subscribe error and configured RPC limits. Done should establish whether the connection limit causes rejected upgrades and define the required behavior or configuration change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.