4paradigm / 4paradigm/OpenMLDB

Zk too many connections when too many online loaddata jobs or too many work threads

オープン
#3,219 コメント 1 件 リアクション 0 件 担当者 1 名 @vagetablechicken が担当を希望しています GitHub で見る
batch-engine bug
主要言語
C++
スター
1.7k
フォーク
331
平均マージ
12日 12時間
マージ済み PR(30日)
1

説明

Too many **load online jobs** create too many connections with zk. And zk default setting `maxClientCnxns=60`.
(Online loaddata will generate too many clients, one zk client for each one.)

If `too many connections` occur, offine job will fail at `fail to init zk handler with hosts`, zk client error is `110(Connection timed out)`, we can't know it's too many connections of zk from offline jobs.

So if you can use diag tool or CLI to connect cluster, but the new offline job will get failed, it may be the zk connections error. To check it, `grep "Too many" `.

If you just run a onebox cluster(taskmanager is local) and just submit one job(no other running jobs), still get the `Too many` error, you might have got it wrong. You should check `show jobs`, no other running jobs in list, and `jps -v` to ensure no orphan offline jobs.

Diag tool can do the checks, it should support:
1. inspect offline --running to get all running jobs
2. execute jps and grep to get all running jobs in system
3. zk log parse(but maybe it’s not allowed to ssh the zk host)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。