containers / containers/podman-compose
Network/resolver not working after update to podman(-compose) 4.2.0 on CentOS Stream 8 (while running Mastodon)
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 622
- PR merge metrics
- No merged PRs in 30d
Description
My Mastodon container stopped working after an update to Podman version 4.2.0 (on CentOS Stream 8), and I haven't found a way to get it up and running again. My impression now is that this problem is caused by the network/resolving not working.
Using 'podman-compose up -d' (as root), all five containers are started, but all three containers mastodon_web_1, mastodon_streaming_1 and mastodon_sidekiq_1 remain in a unhealthy/restarting->highload loop because they are not able to reach the two back-end containers mastodon_db_1 and mastodon_redis_1. The back-end containers themselves are running fine though.
Looking at the logs I see that all three containers cannot reach the db and redis containers, producing stuff like this:
`/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in ``rescue in new_client': could not translate host name "db" to address: Name or service not known (ActiveRecord::ConnectionNotEstablished)`
`/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg.rb:58:in ``initialize': could not translate host name "mastodon_db_1" to address: Name or service not known (PG::ConnectionBad)`
`2022-11-21T13:20:36.078Z pid=7 tid=bin WARN: ActiveRecord::ConnectionNotEstablished: could not translate host name "db" to address: Name or service not known`
All messages in the logs of the depending containers seem to point to a network/resolver issue.
Here is what I tried and found:
- set `DB_HOST=db` (was `DB_HOST=/var/run/postgresql`) and `REDIS_HOST=redis` (was `REDIS_HOST=localhost`);
this was my setup before and updated and worked fine
- set `DB_HOST=mastodon_db_1` and `REDIS_HOST=mastodon_redis_1`
- set `DB_HOST=localhost` and `REDIS_HOST=localhost`
not working (connection issues), even though it's my understanding that containers in the same pod are all reachable through localhost
- outcommented the network configuration in docker-compose.yml (per several online suggestions of people having similar issues)
- outcommented only the `internal: true` statement in docker-compose.yml
- check that dnsname is available in /usr/libexec/cni/
- check: `dnf install containernetworking-plugins: already installed`
- changed network to netavark: in /usr/share/containers/containers.conf: `network_backend = "netavark"` [was: cni]
- check: `dnf install netavark aardvark-dns: already installed`
None of this brought me any further. No idea what else I can try, so any help/hints would be greatly appreciated.
The startup sequence:
```
[root@services mastodon]# podman-compose up -d
['podman', '--version', '']
using podman version: 4.2.0
** excluding: set()
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'tootsuite/mastodon']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_db_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=db -e POSTGRES_HOST_AUTH_METHOD=trust -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/postgres14:/var/lib/postgresql/data:z --net mastodon_internal_network --network-alias db --shm-size 256mb --restart always --healthcheck-command /bin/sh -c pg_isready' '-U' 'postgres postgres:14-alpine
c8e71250cdcf644b2d5d172ffe1045cff91131354951d6aae4d8acea99e344c0
exit code: 0
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_redis_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=redis -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/redis:/data:z --net mastodon_internal_network --network-alias redis --restart always --healthcheck-command /bin/sh -c redis-cli' 'ping redis:7-alpine
4c90f1c2c4024e14d50750adc093d27e1c4e4e8bf18166b7a537efddca16c0c2
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_web_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=web --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/public/system:/opt/mastodon/public/system:z --net mastodon_external_network,mastodon_internal_network --network-alias web -p 127.0.0.1:3000:3000 --restart always --healthcheck-command /bin/sh -c 'wget -q --spider --proxy=off localhost:3000/health || exit 1' tootsuite/mastodon bash -c rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000
319b55314106eb77e4f2bca419cd5311e1db825f1f461901702bc96e9e2ef3ce
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_streaming_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=streaming --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production --net mastodon_external_network,mastodon_internal_network --network-alias streaming -p 127.0.0.1:4000:4000 --restart always --healthcheck-command /bin/sh -c 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1' tootsuite/mastodon node ./streaming
f3a3a7c1fc93e1939f0917be4f27824c0a02f8bef3ec2761f4321c000a1e7944
exit code: 0
['podman', 'network', 'exists', 'mastodon_external_network']
['podman', 'network', 'exists', 'mastodon_internal_network']
podman run --name=mastodon_sidekiq_1 -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=mastodon --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=mastodon --label com.docker.compose.project.working_dir=/home/oc-mastodon/podman-mastodon.offerman.com/mastodon --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=sidekiq --env-file /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/.env.production -v /home/oc-mastodon/podman-mastodon.offerman.com/mastodon/docker-volume/public/system:/opt/mastodon/public/system:z --net mastodon_external_network,mastodon_internal_network --network-alias sidekiq --restart always --healthcheck-command /bin/sh -c 'ps aux | grep '"'"'[s]idekiq 6'"'"' || false' tootsuite/mastodon bundle exec sidekiq
b83545932e550f2722949aabf0813685b3b2655ed1873cf8656a47b4cc702429
exit code: 0
```
[docker-compose.yml.txt](https://github.com/containers/podman-compose/files/10254054/docker-compose.yml.txt)
Contributor guide
Assessment
This issue has not been assessed yet.