Podman is incompatible with `dockerComposeFile`

Using dockerComposeFile with Podman fails at the updateUID step because it can't find the base image. The result is that VSCode can't start the project in a DevContainer.

This is similar to a recent issue fixed a few months ago.

Background story

In April, this bug was raised, where building a devcontainer would fail during the updateUID step when using Podman.

The identified cause was that during normal operation, Podman expects images to be prefixed, and when they're not, it will ask "what you mean" (unless you have them aliased). As a fallback mechanism, building using Podman will always prefix the images with localhost/ when not previously specified.

The implemented fix for that issue was a condition that checks whether the runtime in use is Podman, and if so, the updateUID process will expect the built image to be prefixed by localhost/.

So what's wrong with it?

While I believe it works well for most cases, in scenarios where we want to have a more sophisticated setup using multiple containers and orchestrating them through docker-compose, it might simply not work.

Using docker-compose and Podman as the runtime, built images are not prefixed by localhost/, instead it's prefixed by docker.io/library just as they would normally be when on Docker runtime. That's a compatibility mechanism, turned on by default, provided by Podman so it could serve as a drop-in replacement, at the cost of being incompatible with itself.

Workarounds

So far I can think of these, from the nicest to the ugliest.

1. Disabling this "compatibility mechanism" I mentioned earlier, thus making every image built by docker-compose use the localhost/ prefix (same behavior as standalone podman build). This can be done by setting the config compat_api_enforce_docker_hub=false on containers.conf
2. Force images to be built using a tag prefixed by localhost/ on the docker-compose build spec.
3. Creating (and using) a shell wrapper around podman that replaces the "podman" static string returned by the podman --version subcommand, so Devcontainers would not think that that binary is podman and therefore won't attempt to prefix the image during the build.

Proposed solution

While a workaround is available, I don't think relying upon a non-default setting is the way to go. I would say that a fix for both cases would be to get the actual resulting image name at the prior step and use it with no heuristic involved. I'm not exactly sure how we could do that, but we can discuss if that's an acceptable (and feasible) option.

Logs

❯ devcontainer up --workspace-folder .
[2 ms] @devcontainers/cli 0.69.0. Node.js v20.17.0. linux 6.10.7-arch1-1 x64.
[182 ms] Start: Run: docker compose -f /home/dpereira/Projects/exploring-devcontainers/reproduce-issue/.devcontainer/compose.yaml --profile * config
[767 ms] Start: Run: docker compose -f /home/dpereira/Projects/exploring-devcontainers/reproduce-issue/.devcontainer/compose.yaml --profile * config
[1236 ms] name: devcontainer
services:
  app:
    build:
      context: /home/dpereira/Projects/exploring-devcontainers/reproduce-issue/.devcontainer
      dockerfile: Dockerfile
    networks:
      default: null
    volumes:
      - type: bind
        source: /home/dpereira/Projects/exploring-devcontainers
        target: /workspaces
        bind:
          create_host_path: true
networks:
  default:
    name: devcontainer_default
[1238 ms] Start: Run: docker inspect --type image mcr.microsoft.com/devcontainers/base:bookworm
[1274 ms] Docker Compose override file for building image:
services:
  app:
    build:
      dockerfile: /tmp/devcontainercli-dpereira/container-features/0.69.0-1725418848091/Dockerfile-with-features
      args:
        - BUILDKIT_INLINE_CACHE=1
        - _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label

[1275 ms] Start: Run: docker compose --project-name reproduce-issue_devcontainer -f /home/dpereira/Projects/exploring-devcontainers/reproduce-issue/.devcontainer/compose.yaml -f /tmp/devcontainercli-dpereira/docker-compose/docker-compose.devcontainer.build-1725418848093.yml build
>>>> Executing external compose provider "/usr/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

Sending build context to Docker daemon     643B
[1/2] STEP 1/1: FROM mcr.microsoft.com/devcontainers/base:bookworm AS dev_container_auto_added_stage_label
--> d033a653a768
[2/2] STEP 1/3: FROM d033a653a7689d966ad7824e7614704e7bb7f3e6f112cedbc2f9b0458d93fcbc AS dev_containers_target_stage
[2/2] STEP 2/3: LABEL devcontainer.metadata="[ {\"id\":\"ghcr.io/devcontainers/features/common-utils:2\"}, {\"id\":\"ghcr.io/devcontainers/features/git:1\"}, {\"remoteUser\":\"vscode\"}, {\"remoteUser\":\"vscode\"} ]"
--> a76ec9135e93
[2/2] STEP 3/3: LABEL "com.docker.compose.image.builder"="classic"
[2/2] COMMIT docker.io/library/reproduce-issue_devcontainer-app:latest
--> ff17c24993ef
[Warning] one or more build args were not consumed: [BUILDKIT_INLINE_CACHE]
Successfully tagged docker.io/library/reproduce-issue_devcontainer-app:latest
ff17c24993efbc6213ef175335e1d5351e4042bc666edee072c63ac67337c87a
Successfully built ff17c24993ef
Successfully tagged reproduce-issue_devcontainer-app
STEP 1/9: FROM localhost/reproduce-issue_devcontainer-app
Trying to pull localhost/reproduce-issue_devcontainer-app:latest...
WARN[0000] Failed, retrying in 2s ... (1/3). Error: initializing source docker://localhost/reproduce-issue_devcontainer-app:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0002] Failed, retrying in 2s ... (2/3). Error: initializing source docker://localhost/reproduce-issue_devcontainer-app:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0004] Failed, retrying in 2s ... (3/3). Error: initializing source docker://localhost/reproduce-issue_devcontainer-app:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: creating build container: initializing source docker://localhost/reproduce-issue_devcontainer-app:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: Command failed: docker build -f /tmp/devcontainercli-dpereira/updateUID.Dockerfile-0.69.0 -t vsc-reproduce-issue-5cc145c441bb31286a6b947c55fd99df4242d54f101a66e6b6a97f7cccc0d9e9-uid --platform linux/amd64 --build-arg BASE_IMAGE=localhost/reproduce-issue_devcontainer-app --build-arg REMOTE_USER=vscode --build-arg NEW_UID=1000 --build-arg NEW_GID=1000 --build-arg IMAGE_USER=root /tmp/devcontainercli-dpereira/empty-folder
    at ctA (/home/dpereira/.config/yarn/global/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:410:3965)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async YtA (/home/dpereira/.config/yarn/global/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:482:3944)
    at async eB (/home/dpereira/.config/yarn/global/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:482:4886)
    at async prA (/home/dpereira/.config/yarn/global/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:663:200)
    at async drA (/home/dpereira/.config/yarn/global/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:662:14706)
{"outcome":"error","message":"Command failed: docker build -f /tmp/devcontainercli-dpereira/updateUID.Dockerfile-0.69.0 -t vsc-reproduce-issue-5cc145c441bb31286a6b947c55fd99df4242d54f101a66e6b6a97f7cccc0d9e9-uid --platform linux/amd64 --build-arg BASE_IMAGE=localhost/reproduce-issue_devcontainer-app --build-arg REMOTE_USER=vscode --build-arg NEW_UID=1000 --build-arg NEW_GID=1000 --build-arg IMAGE_USER=root /tmp/devcontainercli-dpereira/empty-folder","description":"An error occurred setting up the container."}

Info

• VSCode Version: 1.92.2
• Podman version: 5.2.2
• Local OS Version: Arch Linux
• Remote OS Version: N/A
• Remote Extension/Connection Type: Containers

Steps to Reproduce:

Here's a minimal .devcontainers folder for that to be reproducible.

1. Install Podman and docker-compose. On Linux, also enable the podman.sock systemd unit and ensure DOCKER_HOST is set properly.
2. Create a .devcontainer/devcontainer.json with the following content:

   {
     "name": "DevContainer issue",
     "dockerComposeFile": "compose.yaml",
     "service": "app",
     "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
     "remoteUser": "vscode"
   }
   

3. Create a .devcontainer/Dockerfile with the following content:

   FROM mcr.microsoft.com/devcontainers/base:bookworm
   

4. Create a .devcontainer/compose.yml with the following content:

   services:
     app:
       build:
         dockerfile: Dockerfile
       volumes:
         - ../..:/workspaces:cached
   

5. Now try to create the devcontainer using the (using the CLI for convenience):

   $ devcontainer up --workspace-folder .
   

It will probably fail with the error message as described in the logs section above.