containers / containers/podman-compose

Network/resolver not working after update to podman(-compose) 4.2.0 on CentOS Stream 8 (while running Mastodon)

Open
#607 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
622
PR merge metrics
No merged PRs in 30d

Description

My Mastodon container stopped working after an update to Podman version 4.2.0 (on CentOS Stream 8), and I haven't found a way to get it up and running again. My impression now is that this problem is caused by the network/resolving not working.

Using 'podman-compose up -d' (as root), all five containers are started, but all three containers mastodon_web_1, mastodon_streaming_1 and mastodon_sidekiq_1 remain in a unhealthy/restarting->highload loop because they are not able to reach the two back-end containers mastodon_db_1 and mastodon_redis_1. The back-end containers themselves are running fine though.

Looking at the logs I see that all three containers cannot reach the db and redis containers, producing stuff like this:
`/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in ``rescue in new_client': could not translate host name "db" to address: Name or service not known (ActiveRecord::ConnectionNotEstablished)`
`/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg.rb:58:in ``initialize': could not translate host name "mastodon_db_1" to address: Name or service not known (PG::ConnectionBad)`
`2022-11-21T13:20:36.078Z pid=7 tid=bin WARN: ActiveRecord::ConnectionNotEstablished: could not translate host name "db" to address: Name or service not known`

All messages in the logs of the depending containers seem to point to a network/resolver issue.

Here is what I tried and found:
- set `DB_HOST=db` (was `DB_HOST=/var/run/postgresql`) and `REDIS_HOST=redis` (was `REDIS_HOST=localhost`);
this was my setup before and updated and worked fine
- set `DB_HOST=mastodon_db_1` and `REDIS_HOST=mastodon_redis_1`
- set `DB_HOST=localhost` and `REDIS_HOST=localhost`
not working (connection issues), even though it's my understanding that containers in the same pod are all reachable through localhost
- outcommented the network configuration in docker-compose.yml (per several online suggestions of people having similar issues)
- outcommented only the `internal: true` statement in docker-compose.yml
- check that dnsname is available in /usr/libexec/cni/
- check: `dnf install containernetworking-plugins: already installed`
- changed network to netavark: in /usr/share/containers/containers.conf: `network_backend = "netavark"` [was: cni]
- check: `dnf install netavark aardvark-dns: already installed`

None of this brought me any further. No idea what else I can try, so any help/hints would be greatly appreciated.

The startup sequence:

```
[root@services mastodon]# podman-compose up -d
['podman', '--version', '']
using podman version: 4.2.0
** excluding: set()
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_db_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=db -e POSTGRES_HOST_AUTH_METHOD=trust -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/postgres14:/var/lib/postgresql/data:z --net mastodon_internal_network --network-alias db --shm-size 256mb --restart always --healthcheck-command /bin/sh -c pg_isready' '-U' 'postgres postgres:14-alpine
c8e71250cdcf644b2d5d172ffe1045cff91131354951d6aae4d8acea99e344c0
exit code: 0
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_redis_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=redis -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/redis:/data:z --net mastodon_internal_network --network-alias redis --restart always --healthcheck-command /bin/sh -c redis-cli' 'ping redis:7-alpine
4c90f1c2c4024e14d50750adc093d27e1c4e4e8bf18166b7a537efddca16c0c2
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_web_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/public/system:/opt/mastodon/public/system:z --net mastodon_external_network,mastodon_internal_network --network-alias web -p 127.0.0.1:3000:3000 --restart always --healthcheck-command /bin/sh -c 'wget -q --spider --proxy=off localhost:3000/health || exit 1' tootsuite/mastodon bash -c rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000
319b55314106eb77e4f2bca419cd5311e1db825f1f461901702bc96e9e2ef3ce
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_streaming_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=streaming --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production --net mastodon_external_network,mastodon_internal_network --network-alias streaming -p 127.0.0.1:4000:4000 --restart always --healthcheck-command /bin/sh -c 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1' tootsuite/mastodon node ./streaming
f3a3a7c1fc93e1939f0917be4f27824c0a02f8bef3ec2761f4321c000a1e7944
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_sidekiq_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=sidekiq --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/public/system:/opt/mastodon/public/system:z --net mastodon_external_network,mastodon_internal_network --network-alias sidekiq --restart always --healthcheck-command /bin/sh -c 'ps aux | grep '"'"'[s]idekiq 6'"'"' || false' tootsuite/mastodon bundle exec sidekiq
b83545932e550f2722949aabf0813685b3b2655ed1873cf8656a47b4cc702429
exit code: 0
```
[docker-compose.yml.txt](https://github.com/containers/podman-compose/files/10254054/docker-compose.yml.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.