cube-js / cube-js/cube

HTTP/1.1 400 Bad Request when using envoy TLS proxy sidecar to encrypt traffic between cubestore-router and cubestore-worker

Open
#7,367 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

I am deploying cubejs into a kubernetes cluster, as described [here](https://github.com/cube-js/cube/tree/master/examples/kubernetes)

In order to encrypt pod to pod traffic, I am deploying envoy proxy sidecars in each pod to enable TLS, as described [here](https://www.envoyproxy.io/docs/envoy/v1.28.0/intro/arch_overview/security/ssl.html#tls).

This approach works fine for cube-api to cubestore-router pod traffic and cube-refresh-worker to cubestore-router traffic.
Additionally, this approach works for traffic from cube pods to non-cube pods, such as keycloak and ksqldb.

However, this approach fails for traffic between cubestore-router and cubestore-worker. The cubestore-router logs are full or the following errors:

```
2023-11-01 14:25:54,012 ERROR [cubestore::http] Error processing HTTP command: Corrupted message received. Please check your worker and meta connection environment variables.

2023-11-01 14:25:56,550 ERROR [cubestore::util] Error during ChunkProcessing: CubeError { message: "Corrupted message received. Please check your worker and meta connection environment variables.", backtrace: "", cause: User }
2023-11-01 14:25:57,077 ERROR [cubestore::http] Error processing HTTP command: Corrupted message received. Please check your worker and meta connection environment variables.

2023-11-01 14:25:58,169 ERROR [cubestore::scheduler] Error processing event UpdateJob(IdRow { id: 32, row: Job { row_reference: Table(Tables, 17), job_type: TableImportCSV("stream://default/prod_pre_aggregations-k_asset_asset_details_rollup_nxbppwn1_gybfkt1_1ik4nth/0"), last_heart_beat: 2023-11-01T14:25:57.406644645Z, status: ProcessingBy("dbg-cubestore-worker-0.dbg-cubestore-worker-headless:9001") } }, IdRow { id: 32, row: Job { row_reference: Table(Tables, 17), job_type: TableImportCSV("stream://default/prod_pre_aggregations-k_asset_asset_details_rollup_nxbppwn1_gybfkt1_1ik4nth/0"), last_heart_beat: 2023-11-01T14:25:58.164603958Z, status: Error("Stale stream timeout: deadline has elapsed") } }): Corrupted message received. Please check your worker and meta connection environment variables.
2023-11-01 14:26:00,165 ERROR [cubestore::http] Error processing HTTP command: Corrupted message received. Please check your worker and meta connection environment variables.
```
The envoy logs show the following error:
`HTTP/1.1 400 DPE 0 11 http1.codec_error HPE_INVALID_METHOD `
(DPE = Downstream Protocol Error. Downstream = cubestore-router).

Attached is a screenshot of a tcpdump of traffic, if it sheds any light on what is going on..
![Screenshot 2023-10-24 at 12 37 46 PM](https://github.com/cube-js/cube/assets/2110426/55c279c3-e14e-43f2-b476-b1964d730fff)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.