graphql-hive / graphql-hive/console
Usage/Usage-ingestor container cant deal with restart of 'broker'
- Dominant language
- TypeScript
- Stars
- 483
- Forks
- 145
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 65
Description
We're self hosting hive through kubernetes. In kubernetes each pod (container) can be restarted at any time.
The concept of 'depends_on:' as defined in the docker-compose doesnt exist within kubernetes landscape. Each container should be able to deal with a restart of it's dependencies. This works nicely for all containers except for the beforementioned usage/usage-ingestor.
For usage this leads to the following logs:
```
"msg":"[503] (::ffff:127.0.0.6) POST / (reqId=f74caede-fb53-4c47-a0fb-d2c7c720b657)"}
"msg":"Not ready to collect report (token=989••••••••••••••••••••••••••fd7)"}
```
For the usage-ingestor pod the logs look a bit more extensive:
```
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":4,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401577124}
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":5,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401577507}
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":6,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401578130}
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":7,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401579163}
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":8,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401581205}
"broker":"hive-broker-svce:29092","clientId":"usage-ingestor","error":"The group coordinator is not available","correlationId":9,"size":55,"msg":"[Connection] Response GroupCoordinator(key: 10, version: 2)","time":1705401584505}
"groupId":"usage-ingestor-v2","stack":"KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator\n at Cluster.findGroupCoordinatorMetadata (file:///usr/src/app/index.js:74709:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async file:///usr/src/app/index.js:74644:37\n at async [private:ConsumerGroup:join] (file:///usr/src/app/index.js:76901:28)\n at async file:///usr/src/app/index.js:77034:13\n at async Runner.start (file:///usr/src/app/index.js:77694:11)\n at async start (file:///usr/src/app/in.js:78348:11)","msg":"[Consumer] Crash: KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator","time":1705401584506}
Consumer stopped
[Consumer] Stopped","time":170540158450
Consumer disconnected
Consumer crashed (restart=false, error=KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator)
Restarting consumer...
Starting Usage Ingestor...
Connecting Kafka Consumer
Subscribing to Kafka topic: usage_reports_v2
broker":"hive-broker-svce:29092","clientId":"usage-ingestor","stack":"Error [ERR_STREAM_WRITE_AFTER_END]: write after end\n at new NodeError (node:internal/errors:405:5)\n at _write (node:internal/streams/writable:322:11)\n at Writable.write (node:internal/streams/writable:337:10)\n at Object.sendRequest (file:///usr/src/app/index.js:74088:31)\n at SocketRequest.send [as sendRequest] (file:///usr/src/app/index.js:72825:27)\n at SocketRequest.send (file:///usr/src/app/index.js:72644:14)\n at RequestQueue.sendSocketRequest (file:///usr/src/app/index.js:72865:23)\n at RequestQueue.push (file:///usr/src/app/index.js:72849:16)\n at file:///usr/src/app/index.js:74083:33\n at new Promise ()","msg":"[Connection] Connection error: write after end","time":1705401584509}
{"level":50,"time":1705401584510,"pid":14,"hostname":"hive-usage-deployment-6d75fc64b-7qqzz","logger":"kafkajs","eventName":"consumer.crash","stack":"KafkaJSConnectionError: Connection error: write after end\n at Socket.onError (file:///usr/src/app/index.js:73919:27)\n at Socket.emit (node:events:517:28)\n at Socket.emit (node:domain:489:12)\n at emitErrorNT (node:internal/streams/destroy:151:8)\n at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","msg":"[Consumer] Failed to execute listener: Connection error: write after end","time":1705401584510}
```
Reproduction:
- Start up all of the containers
- Stop/start the broker container
- check the logs of usage/usage-ingestor to find the errors above
Expected solution:
- The usage/usage-ingestor container should be able to restart at any time without cause the processing/ingestion of before mentioned containers to stop working
Contributor guide
Assessment
This issue has not been assessed yet.