Airbyte local installation on fresh linux installation fails with connection refused errors on temporal pod -- root cause is db connection
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Helm Chart Version
2.2.0
### What step the error happened?
On deploy
### Relevant information
Enviorment:
- Host: RHEL 9
- Kernel: 5.14.0-687.36.1.el9_8.x86_64
- Docker: 29.7.2
- abctl: v0.30.4
Executing local installation will lead to following error after some time:
```
ERROR i.m.r.Micronaut(handleStartupException):362 - Error starting an of type [io.airbyte.commons.temporal.TemporalInitializationUtils]
...
Message: Could not create Temporal client within max timeout!
...
Caused by: java.lang.RuntimeException: Could not create Temporal client within
```
Checking the kubernet cluster shows several pods crashing and restarting, mainly because temporal is not reachable. My knowledge of kubernet and the airbyte setup gets a bit weak here, so hopefully i get all information right.
Within the temporal pod the log-output is:
`nc: bad address 'airbyte-db-svc.airbyte-abctl.svc.cluster.local'
Waiting for PostgreSQL to startup.`
But the database pod is running and accepts connection.
Some test (with the help of AI) shows that the temporal pod is unable to resolve the address of the db pod.
`getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local
→ exit code 2`
Comparing the dns setup of the temporal pod with the manifest server shows no differences.
Testing the dns resolver with a new temporal pod and an alpine image shows the same issue.
`kubectl run temporal-dns-test -n airbyte-abctl --image=temporalio/auto-setup:1.27.2 --restart=Never --command -- sleep 300`
`kubectl exec -n airbyte-abctl temporal-dns-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed
`
`kubectl run alpine-test -n airbyte-abctl --image=alpine:3.21 --restart=Never --command -- sleep 300`
`kubectl exec -n airbyte-abctl alpine-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed
At the end I was able to start airbyte with the following patch which change the alpine based pods dns ndots setting to "1" :
```
kubectl patch deployment airbyte-abctl-temporal -n airbyte-abctl --type='strategic' -p '{
"spec": {
"template": {
"spec": {
"dnsConfig": {
"options": [
{
"name": "ndots",
"value": "1"
}
]
}
}
}
}
}'
```
Directly after this the other pods were able to start successful.
I have understood that this is more like a temporary workaround which will be lost the next time I use abctl local install.
For now I would like to share my experience so that it can be solved for others as well.
If there is an easy fix for it, maybe it would help to be included in the installation guide.
### Relevant log output
```shell
```
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13313
貢獻指南
研究方向
Start by reproducing the fresh RHEL 9 installation with Helm chart 2.2.0 and abctl v0.30.4, then inspect the airbyte-abctl-temporal deployment and airbyte-db-svc DNS resolution. Compare the failing pods with the Alpine test and verify the ndots workaround; done means local installation starts without Temporal connection-refused errors and the fix survives a new abctl install.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- docker, helm, kubernetes, linux, postgresql
- 領域
- databases, devops, infrastructure
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100