airbytehq / airbytehq/airbyte

Airbyte local installation on fresh linux installation fails with connection refused errors on temporal pod -- root cause is db connection

未關閉
#84,412 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Helm Chart Version

2.2.0

### What step the error happened?

On deploy

### Relevant information

Enviorment:

- Host: RHEL 9
- Kernel: 5.14.0-687.36.1.el9_8.x86_64
- Docker: 29.7.2
- abctl: v0.30.4

Executing local installation will lead to following error after some time:

```
ERROR i.m.r.Micronaut(handleStartupException):362 - Error starting an of type [io.airbyte.commons.temporal.TemporalInitializationUtils]
...
Message: Could not create Temporal client within max timeout!
...
Caused by: java.lang.RuntimeException: Could not create Temporal client within
```

Checking the kubernet cluster shows several pods crashing and restarting, mainly because temporal is not reachable. My knowledge of kubernet and the airbyte setup gets a bit weak here, so hopefully i get all information right.

Within the temporal pod the log-output is:

`nc: bad address 'airbyte-db-svc.airbyte-abctl.svc.cluster.local'
Waiting for PostgreSQL to startup.`

But the database pod is running and accepts connection.

Some test (with the help of AI) shows that the temporal pod is unable to resolve the address of the db pod.

`getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local
→ exit code 2`

Comparing the dns setup of the temporal pod with the manifest server shows no differences.

Testing the dns resolver with a new temporal pod and an alpine image shows the same issue.

`kubectl run temporal-dns-test -n airbyte-abctl --image=temporalio/auto-setup:1.27.2 --restart=Never --command -- sleep 300`

`kubectl exec -n airbyte-abctl temporal-dns-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed
`

`kubectl run alpine-test -n airbyte-abctl --image=alpine:3.21 --restart=Never --command -- sleep 300`

`kubectl exec -n airbyte-abctl alpine-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed

At the end I was able to start airbyte with the following patch which change the alpine based pods dns ndots setting to "1" :

```
kubectl patch deployment airbyte-abctl-temporal -n airbyte-abctl --type='strategic' -p '{
"spec": {
"template": {
"spec": {
"dnsConfig": {
"options": [
{
"name": "ndots",
"value": "1"
}
]
}
}
}
}
}'
```

Directly after this the other pods were able to start successful.

I have understood that this is more like a temporary workaround which will be lost the next time I use abctl local install.

For now I would like to share my experience so that it can be solved for others as well.

If there is an easy fix for it, maybe it would help to be included in the installation guide.

### Relevant log output

```shell

```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13313

貢獻指南

開啟貢獻指南

研究方向

Start by reproducing the fresh RHEL 9 installation with Helm chart 2.2.0 and abctl v0.30.4, then inspect the airbyte-abctl-temporal deployment and airbyte-db-svc DNS resolution. Compare the failing pods with the Alpine test and verify the ndots workaround; done means local installation starts without Temporal connection-refused errors and the fix survives a new abctl install.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
docker, helm, kubernetes, linux, postgresql
領域
databases, devops, infrastructure
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。