containers / containers/podman-compose

aardvark-dns exits during `podman-compose up` for 4-6 service projects on Windows/WSL2 — does NOT reproduce with raw `podman run` (incl. parallel)

Open
#1,463 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
6.2k
Forks
622
PR merge metrics
No merged PRs in 30d

Description

Summary


podman-compose up consistently fails for a specific class of multi-container projects with:


Error: unable to start container "<id>": netavark (exit code 1): error while applying dns entries: aardvark-dns exited unexpectedly without error message

Affected containers stay stuck in Created. The crash leaves no aardvark-dns log line — netavark's wrapper message is the only signal.


Crucially: I cannot reproduce this with the podman CLI alone. A standalone reproducer that issues the equivalent podman network create + podman run calls — both sequentially and concurrently — succeeds every time on the same machine. The failure only appears when podman-compose drives the bring-up, which is why I'm filing here rather than at containers/podman. (Originally reported at containers/podman#<original-issue-number>; a maintainer confirmed podman-compose is out of scope there and asked for a raw-CLI repro, which is included below.)


Environment



  • Host OS: Windows 10 Pro 25H2 build 26200.8457

  • podman: 5.8.2 (WSL2 backend)

  • podman-compose: version 1.5.0

    [repro-logs.zip](https://github.com/user-attachments/files/28081488/repro-logs.zip)
    [repro-logs-parallel.zip](https://github.com/user-attachments/files/28081487/repro-logs-parallel.zip)

  • Network backend: netavark

  • DNS: aardvark-dns (bundled with podman 5.x)

  • Rootless: yes


Minimal reproduction


repro-compose.yaml:


yaml
name: repro-aardvark-bug

services:
postgres:
image: docker.io/postgres:16.13-alpine
environment:
POSTGRES_USER: app
POSTGRES_PASSWORD: app
POSTGRES_DB: app
clickhouse:
image: docker.io/clickhouse/clickhouse-server:26.3
environment:
CLICKHOUSE_USER: app
CLICKHOUSE_PASSWORD: app
redis:
image: docker.io/library/redis:7-alpine
minio:
image: docker.io/minio/minio:latest
command: ["server", "/data"]
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin

podman-compose -f repro-compose.yaml up -d

(From a clean state: podman stop -a; podman rm -af; podman volume prune -f; podman network prune -f.)


Observed



  • postgres + clickhouse reach Up; the remaining services fail to start with the aardvark-dns exited unexpectedly error and stay Created.

  • Retrying podman-compose up, tearing down and re-running, host reboot, and full podman machine rm -f + init + start all produce the same failure.

  • The failure clusters on 4-6 service projects that include bare postgres + clickhouse service names. Single-container and 3-service projects work; 10+ service projects with prefixed service names (e.g. lf-postgres) also work.


Expected


All services reach Up; aardvark-dns registers DNS entries for every service on the project's default network.


Key diagnostic — raw podman CLI does NOT reproduce it


I built a reproducer using only podman CLI (no podman-compose), --log-level=debug on every call, fresh custom network per test, full teardown between tests. 8 of 8 tests pass, including the exact shapes that fail under podman-compose:



Test | Shape | Result
-- | -- | --
1× alpine, default net | sanity | PASS
1× alpine, custom net | sanity | PASS
postgres + clickhouse (bare names) | 2-svc | PASS
postgres + clickhouse + redis + minio (bare) | 4-svc | PASS
+ minio-init (matches a real failing compose) | 5-svc | PASS
same 4-svc, prefixed names (lf-*) | control | PASS
4-svc bare, started in parallel | concurrency | PASS
4-svc bare, parallel, ×5 back-to-back | flake-hunt | PASS


The parallel tests fire 4 concurrent podman run -d calls into the same fresh network (4-5 aardvark registrations inside a ~1s window) and still succeed. So the trigger is not raw podman run, bare service names, service count, or concurrent DNS registration in isolation — it's something podman-compose layers on top of podman run.


What podman-compose does that the raw-CLI repro did not


Candidate triggers, for whoever investigates:



  1. Healthchecks — compose healthcheck: adds --health-cmd/--health-interval; healthcheck threads do in-container DNS lookups.

  2. Bind-mount volumes — host-path mounts (on WSL2 these route through 9p; first-time setup can race with container start).

  3. Port publishing-p for every exposed port.

  4. Restart policies--restart=unless-stopped (restart loops re-register DNS for the same name).

  5. --env-file rather than per--e.

  6. Container labelsio.podman.compose.*.

  7. Network creation options — any --opt / --subnet / --ipam-driver podman-compose passes when creating the project's default network.

  8. Whether podman-compose uses podman play kube vs direct podman run for any path.


I'm happy to extend the CLI reproducer with any one of these (healthchecks + bind-mounts + ports + restart are present on every failing project) to bisect which one trips aardvark — just let me know which you'd like first, or whether a --log-level=debug trace of the actual failing podman-compose up (showing the exact podman invocations issued) would be more useful.


What I tried (none fixed it)



  • Full Windows reboot.

  • podman machine stop/start.

  • podman machine rm -f + init + start (verified fresh via podman info).

  • podman system prune -af --volumes; podman network prune -f between attempts.

  • Manual podman network create + container-on-network — works standalone.

  • DNS warmup (create+destroy a network with one container before the failing compose) and a single retry-after-warmup — both still fail under compose.


Attachments



  • repro-logs.zip / repro-logs-parallel.zip — full --log-level=debug output per step from the raw-CLI reproducer (all PASS), plus podman info and podman --version.

  • A --log-level=debug capture of the failing podman-compose up can be provided on request.


Diagnostic-improvement note


aardvark-dns crashes "without error message" — the only signal is netavark's exit-code-1 wrapper. A more verbose aardvark-dns log mode, or capturing the aardvark-dns subprocess stderr into the podman/compose log, would make this class of issue much easier to diagnose.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.