airbytehq / airbytehq/airbyte

Airbyte local installation on fresh linux installation fails with connection refused errors on temporal pod -- root cause is db connection

Ouverte
#84,412 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Helm Chart Version

2.2.0

### What step the error happened?

On deploy

### Relevant information

Enviorment:

- Host: RHEL 9
- Kernel: 5.14.0-687.36.1.el9_8.x86_64
- Docker: 29.7.2
- abctl: v0.30.4

Executing local installation will lead to following error after some time:

```
ERROR i.m.r.Micronaut(handleStartupException):362 - Error starting an of type [io.airbyte.commons.temporal.TemporalInitializationUtils]
...
Message: Could not create Temporal client within max timeout!
...
Caused by: java.lang.RuntimeException: Could not create Temporal client within
```

Checking the kubernet cluster shows several pods crashing and restarting, mainly because temporal is not reachable. My knowledge of kubernet and the airbyte setup gets a bit weak here, so hopefully i get all information right.

Within the temporal pod the log-output is:

`nc: bad address 'airbyte-db-svc.airbyte-abctl.svc.cluster.local'
Waiting for PostgreSQL to startup.`

But the database pod is running and accepts connection.

Some test (with the help of AI) shows that the temporal pod is unable to resolve the address of the db pod.

`getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local
→ exit code 2`

Comparing the dns setup of the temporal pod with the manifest server shows no differences.

Testing the dns resolver with a new temporal pod and an alpine image shows the same issue.

`kubectl run temporal-dns-test -n airbyte-abctl --image=temporalio/auto-setup:1.27.2 --restart=Never --command -- sleep 300`

`kubectl exec -n airbyte-abctl temporal-dns-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed
`

`kubectl run alpine-test -n airbyte-abctl --image=alpine:3.21 --restart=Never --command -- sleep 300`

`kubectl exec -n airbyte-abctl alpine-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local`
-> Failed

At the end I was able to start airbyte with the following patch which change the alpine based pods dns ndots setting to "1" :

```
kubectl patch deployment airbyte-abctl-temporal -n airbyte-abctl --type='strategic' -p '{
"spec": {
"template": {
"spec": {
"dnsConfig": {
"options": [
{
"name": "ndots",
"value": "1"
}
]
}
}
}
}
}'
```

Directly after this the other pods were able to start successful.

I have understood that this is more like a temporary workaround which will be lost the next time I use abctl local install.

For now I would like to share my experience so that it can be solved for others as well.

If there is an easy fix for it, maybe it would help to be included in the installation guide.

### Relevant log output

```shell

```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13313

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.