cube-js / cube-js/cube

[cubestore] Websocket ResetWithoutClosingHandshake on router causes RWLoop channel closed error in ECS Fargate deployment

Open
#10,541 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

## Describe the bug

The cubestore router produces a `Websocket error: Protocol(ResetWithoutClosingHandshake)` followed immediately by a `Create table failed: Row with id 1 is not found for TableRocksTable` error. This causes the RWLoop channel to close, resulting in the Cube API throwing:

```
ERROR [cubestore::cluster::ingestion::job_runner]
Error in processing loop: Internal: Error during scheduling
write task in loop: Failed to schedule task to RWLoop
(default), error: channel closed
```

This error recurs periodically and causes the dashboard to stop functioning until the tasks are manually restarted.

---

## Environment

- **CubeStore version**: v1.6.1
- **Cube API version**: latest (custom ECR image)
- **Deployment**: AWS ECS Fargate
- **Networking**: ECS Service Connect (not NLB for internal services)
- **Database**: MySQL (AWS RDS)
- **Pre-aggregation storage**: AWS S3

## Architecture

- 1x Cube API task (cpu: 1024, memory: 2048)
- 1x CubeStore Router task (cpu: 1024, memory: 2048)
- 2x CubeStore Worker tasks (cpu: 2048, memory: 4096)
- Router connected via: `CUBEJS_CUBESTORE_HOST=router:9999`

---

## Exact error sequence from CloudWatch logs (cubestore router)

```
2026-03-23T18:16:12Z INFO [cubestored] Cube Store version 1.6.1
2026-03-23T18:16:42Z ERROR [cubestore::mysql] Error during
processing MySQL connection: peer terminated connection

[MySQL connection reset errors continue every 30s]

2026-03-23T18:18:11Z ERROR [cubestore::http] Websocket error:
Protocol(ResetWithoutClosingHandshake)

2026-03-23T20:54:12Z ERROR [cubestore::http] Websocket error:
Protocol(ResetWithoutClosingHandshake)

2026-03-23T20:54:12Z ERROR [cubestore::http] Error processing
HTTP command: Create table failed: Row with id 1 is not
found for TableRocksTable
```

---

## Resource metrics at time of error (confirmed from CloudWatch)

Worker-1 at time of errors:
- Memory: **63 MB used / 512 MB reserved (12.3%)**
- CPU: **0.74 units / 256 reserved (0.29%)**

Router CPU: spikes to **94.48%** during/after the WebSocket error, then drops.

Workers: **zero errors** in logs. All jobs completing successfully.

**OOM is not the cause. Resources are not the bottleneck.**

---

## What we have already tried

1. Increased all ECS task resources (cpu 2048/4096, memory 2048/4096) — error persists
2. Restarting ECS services — error returns after some time
3. Checked RDS proxy logs — no MySQL-level disconnect from the DB side

---

## Questions

1. What causes `Websocket error: Protocol(ResetWithoutClosingHandshake)` on the router in an ECS Fargate + Service Connect deployment?
2. Is `Row with id 1 is not found for TableRocksTable` a consequence of the WebSocket drop, or an independent cause?
3. Does v1.6.1 have any known issues with router WebSocket stability in containerised deployments?
4. Is there a recommended way to recover from metastore inconsistency without clearing the S3 state?

---

## Additional context

- MySQL errors every 30 seconds appear to be ECS health check probes hitting CubeStore's MySQL wire protocol port (3306) — we believe this is noise, not the root cause
- The error first occurred on 2026-03-02 and again on 2026-03-09, both on Mondays, suggesting it may be triggered by scheduled pre-aggregation refresh load
- Workers show `NodeInMemoryChunksCompaction` and `TableImportCSV` jobs completing successfully throughout the incident

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.